From Sync APIs to Task-Based Inference: Priority, Standard, Flex, Async, and Batch

Over the past few years, the biggest win of the LLM API has been turning complex GPU inference into one very simple call:

The application sends a request, the model computes, the HTTP connection stays open, and a response comes back. That interface fits chatbots, copilots, and search Q&A very well.

In the agent era, a problem is getting harder to ignore: not every model call deserves to be executed the same way.

• A chat request with a user waiting on it may care a lot about whether the first token arrives in 500 ms or 3 seconds.

• A code analysis step inside a coding agent can run tens of seconds slower without anyone noticing.

• A deep research task that runs for ten minutes probably should not hold an HTTP connection open the whole time.

• An evaluation over 100,000 samples may not belong in 100,000 separate API requests at all. It may be better submitted as one batch job.

These workloads may all use exactly the same model. What differs is the latency requirement, the execution model, and the request scale.

So LLM infrastructure is moving away from a single Request → Response shape toward three layers of questions:

1. How urgent is this synchronous request? Priority, Standard, or Flex.

2. Should this task still hold a synchronous connection? Sync or Async.

3. Is this one request, or a dataset? Single task or Batch.

Understanding these three questions is the key to understanding the next generation of inference infrastructure.

1. Why can "a bit slower" be cheaper?

Under the hood, LLM inference is still a GPU scheduling problem. Online inference usually optimizes for:

• TTFT (time to first token)

• TPOT / TBT (per-token generation latency)

• P95 / P99 latency

To run requests as fast as possible, an inference platform has to keep GPU headroom in reserve, control queue length, admit requests quickly, avoid waiting for bigger batches to form, and stand up extra capacity for traffic peaks.

All of that costs money. Say there are only two requests, A and B. To minimize latency, the scheduler may start them immediately.

But if the application tells the scheduler "I can wait 30 seconds", the scheduler can keep collecting requests C through N, then form larger batches, raise GPU utilization, use off-peak and spare capacity, and look for cheaper resources across more regions and providers.

From a compute economics point of view, latency tolerance is itself a resource. The more time an application is willing to give up, the more scheduling freedom the infrastructure has.

2. Service tiers answer the first question: how urgent is a synchronous request?

Google Gemini already makes this distinction explicit with Priority, Standard, Flex, and Batch. The first three are synchronous interfaces. Flex trades higher latency for lower cost by running on lower-priority, preemptible capacity. Batch is the truly asynchronous bulk interface. Google describes Flex latency targets in minutes, and Batch completion windows of up to 24 hours.

The most important point here: Priority, Standard, and Flex change the scheduling policy, not the programming model. The application still sends a request over an HTTP connection and gets a response back. The only difference is which compute lane the request enters.

3. Priority: paying a premium for latency and predictability

Priority fits workloads where a user is waiting and latency carries real business value: chat, search answers, voice agents, interactive copilots, customer-facing agents, and core online decision APIs.

What it buys is less queueing, steadier latency, and better capacity during peaks. So Priority should not be read as "the more expensive API". A more accurate reading is: you pay a premium for more predictable real-time capacity.

If a request uses only a few thousand tokens, the extra amount paid for Priority can be very small in absolute terms. Cutting TTFT from 4 seconds to 1 second, on the other hand, can be very noticeable to the end user.

4. Standard: the balance point for most production workloads

Standard usually carries the bulk of production traffic. It balances cost, latency, and reliability. Typical examples are everyday chat, document QA, agent planners, general API automation, and code explanation.

These need a reasonably fast response, but not badly enough to justify Priority capacity. Standard should usually be the default production tier.

5. Flex: cheaper, but still a synchronous API

Flex is the mode most often confused with Async.

On Infron today, Flex is still synchronous inference. The call looks the same: the application sends a request to Infron, Infron waits for lower-priority capacity, the model runs, and the response comes back. What changes is that price goes down, scheduling priority goes down, and expected latency goes up.

Infron's model marketplace also defines Flex, Standard, and Priority as separate provider pools. The model and its output capabilities are the same, while stability, speed, and price differ. For some models, the Flex price is roughly half of Standard.

That makes Flex a good fit for a case that used to be poorly served: "I want to keep using the normal API, but this request is not urgent." For example:

• Offline evals

• Development

• Background classification

• Data enrichment

• Internal analytics

• Non-critical steps inside an agent

Its biggest advantage is low migration cost. Application code does not need to become a job system. The request is simply sent to a lower-priority lane.

Synchronous Flex does have built-in engineering trade-offs, though. If a Flex request waits a long time for capacity, the model bill is fine, but the client is still holding an HTTP connection open while it waits. That brings its own set of problems:

• Long HTTP timeouts

• Connection resets

• Retries

• Occupied workers

• Serverless compute cost

• Workflow stability

Suppose a workflow runtime bills by worker time and a model request sits in the queue for 5 minutes. The model got cheaper, but the application worker also sat idle for 5 minutes. The total cost is not necessarily lower.

6. Async is not about price. It is about the task lifecycle.

The core change in async inference is this. A traditional API goes submit, wait, wait, wait, response. Async goes:

For the first time, model inference stops being an HTTP request and becomes a real job. The application no longer has to be tied to how long the model takes.

In Infron's async mode, the application submits an async inference task. Once Infron accepts it, the task moves through its own states:

The application no longer has to keep the original synchronous HTTP request alive. When the task finishes, the result is retrieved through the async task mechanism.

Architecturally this is a significant shift. Synchronous inference manages a request lifecycle. Async manages a task lifecycle. That means the infrastructure has to start dealing with task IDs, task state, persistence, retries, timeouts, cancellation, result storage, idempotency, and observability.

This is no longer a reverse proxy problem. The infrastructure is moving into workflow runtime territory.

7. Where does Async fit best?

A good test is: does the user actually need to keep waiting on this model call?

Take a coding agent. The task might go plan, read the repository, analyze, edit, run tests, review, and take 5, 10, or 20 minutes end to end. The user usually cares about when the whole task will be done, not about which second the 17th LLM call returned.

That workload fits a submit task, suspend workflow, async inference, resume pattern very well, especially on runtimes that are already good at suspend, wait, and resume: Temporal, Trigger.dev, Inngest, Airflow, Celery, and serverless workflow engines.

8. The economic value of Async is often underestimated

People often ask how much cheaper an async API is than a sync one. That may not be the most important question.

What Async saves first is often not model token cost but application waiting cost.

For agents and workflows, the number that matters is:

Inference cost + application compute + orchestration + retries

not just $/1M tokens.

9. How is Batch different from Async?

This is another common source of confusion. Async usually handles one task. Batch handles a dataset. Both run asynchronously, but at completely different granularity.

Async maps one request to one task to one result. It fits agents, PR review, deep research, long-running reasoning, and background generation. Requests may depend on each other in sequence: task B needs the result of task A.

Batch takes a JSON or JSONL file containing requests 1 through 100,000, runs it as one batch job, and produces a result file. It fits evaluation, embeddings, synthetic data, classification, ETL, structured extraction, image processing, and re-indexing. These requests are usually independent of each other.

10. Batch is a file-level API

The basic unit of a traditional API is a request. The basic unit of Async is a task. The basic unit of Batch is a file or dataset.

Instead of sending 10,000 HTTP calls, you prepare a batch.jsonl:

Then: upload, create the batch, processing, batch completed, download results.

OpenAI's Batch API is a typical file-oriented job: upload a JSONL file, create a batch, and match each result back to its request by custom_id. AWS Bedrock follows a similar model: the JSONL data goes to S3, you create a model invocation job, and the results are written back to an output location.

Batch has settled into a fairly consistent industry pattern.

11. A taxonomy for next-generation inference

A structure that matches how products and teams actually use inference today looks like this:

Each of the five modes optimizes something different:

The thing worth noticing in this table: Flex and Async are not substitutes for each other. Flex optimizes the scheduling cost of a synchronous request. Async optimizes the execution model of a long-running task. Batch optimizes throughput for large volumes of independent requests.

12. How to decide which mode a workload should use

Start with three questions.

Question 1: Is a user waiting right now?

If yes, start with the synchronous interface, then pick by latency requirement: latency-critical goes to Priority, normal production goes to Standard, latency-tolerant goes to Flex.

Question 2: Is this a long-running task?

Agent tasks, deep research, PR review, complex coding, and video generation can run for several minutes or longer. That is when to start considering Async. The real question is not whether it takes 30 seconds or 5 minutes. It is whether the application needs to hold a connection open while it waits.

Question 3: Are there lots of independent requests?

If you have 10K, 100K, or 1M independent records, look at Batch. At that scale, throughput matters far more than single-request latency.

A practical decision tree:

It is not a strict algorithm, but for AI infrastructure teams it is a useful first-pass classification.

13. Agents are the workload most worth redesigning

What makes agents special is that one user request fans out into many LLM calls. Behind a single request you might see planner, search, reason, tool, reason, code, test, review, fix: dozens or even hundreds of model calls.

Running the whole agent on synchronous Standard is the simplest implementation, but not necessarily the most economical.

At Infron, our view is that an agent should first separate out its critical path:

Some calls directly shape the user experience: the planner, the coder, the final answer. Others are background work: research, verification, memory extraction, evaluation. That naturally splits into:

• Critical path → Sync

• Long-running background work → Async

• Massive independent work → Batch

This is a more sensible approach than simply switching every call to a cheaper model. Here is each path in more detail.

For agents, Flex is valuable in sequential workflows

Flex has one advantage Batch cannot offer: it handles sequential dependencies. If LLM call 2 has to see the output of LLM call 1 (with a tool call in between), you cannot generate a batch.jsonl up front and submit everything at once.

Flex responds more slowly but keeps normal synchronous API semantics. Google positions Flex explicitly for non-urgent sequential workflows and names the synchronous call model as the main difference from Batch.

So the steps inside a coding agent that do not need the lowest latency but do depend on earlier steps are a good match for Flex.

For agents, Async is valuable for workflow suspension

If an agent step itself can run for a very long time, say 5 minutes of inference, waiting on it synchronously makes less and less sense. The ideal runtime becomes:

This is the essential boundary between the two: Flex optimizes the scheduler. Async optimizes the workflow.

Batch is the best fit for evaluation

If I had to pick the one workload to move to Batch first, it would be evals.

Say you have 100,000 gateway logs and want to test models A, B, and C. With a normal sync API, that could mean 300,000 API requests. But those requests have no dependencies on each other, so the whole thing can run as:

This is a textbook embarrassingly parallel workload. Whether a single request finishes 2 seconds faster or 20 minutes slower does not matter. What matters is whether the whole dataset finishes before the deadline.

At Infron, we think Batch will change the economics of evaluation. Many teams do not run evals often enough, and it is not for lack of frameworks. It is because inference is expensive and running evals is a hassle. So the common pattern is: right before launch, sample a few hundred records, run it once.

If Batch lowers both the compute cost and the engineering cost, evals can gradually run on every prompt version, every model version, every provider change, and every routing change. That is when evaluation stops being a project and becomes infrastructure.

14. Different modes need different observability

If one gateway serves Priority, Standard, Flex, Async, and Batch at the same time, a single latency dashboard is no longer enough.

• Synchronous APIs: TTFT, TPS, P50, P95, P99, error rate.

• Async: add queue time, task execution time, task completion time, task failure rate, and retry rate.

• Batch: batch size, completed records, failed records, throughput, completion window, and cost per 1K requests.

15. Async and Batch bring new reliability problems

Once an API becomes a job, the infrastructure has to handle things that were less prominent with sync APIs.

Idempotency. If a task submission hits a network timeout, the application does not know whether Infron accepted the task. A naive retry can create a duplicate job.

Partial failure. If a batch finishes with 99,998 successes and 2 failures, the whole batch usually should not be marked failed. You need per-record errors.

Result matching. Batch output generally cannot rely on output order matching input order. Results should be matched by custom_id.

Retry semantics. Transport failures, capacity failures, model failures, and business validation failures each call for a different retry strategy.

At Infron, we think these problems will make inference infrastructure look more and more like a distributed job system, and less like a traditional HTTP proxy.

16. Why the next-generation gateway needs to unify these modes

The genuinely hard part is that providers support these capabilities very unevenly.

Even within one provider, model A may support Flex while model B does not. AWS Bedrock, for example, has regular Invoke, StartAsyncInvoke for some scenarios, and separate batch inference jobs, and Batch requires JSONL input plus an S3 job workflow.

If all of these differences leak into the application, every team has to rebuild provider capability detection, task lifecycle, batch formats, polling, retries, fallback, and result parsing. The benefits of going multi-provider get eaten by integration complexity very quickly.

17. Infron now has three clear inference paths

Seen this way, Infron's current product structure is:

1. Synchronous API. The familiar OpenAI-compatible request/response model, split by SLA into Priority, Standard, and Flex. It answers: how much scheduling priority should this synchronous request get?

2. Async API. Infron turns an inference request into an async task. It answers: this work already takes a long time, so is there any reason to keep the HTTP connection open?

3. Batch API. Large numbers of independent requests go into one JSON / JSONL file and run as a batch job. It answers: how do you process large offline workloads at higher throughput? Infron's public Batch capability already handles offline async inference as batch jobs, with lifecycle endpoints for job status and results.

18. What Infron optimizes is more than token price

The industry is used to comparing $/1M tokens. As Flex, Async, and Batch become common, that metric becomes more and more incomplete, because the real cost of an agent task is:

Inference + application compute + orchestration + retries + failures

A model with very cheap tokens that ties up a worker for 10 minutes is not necessarily cheap. A batch where each individual request is slow, but 100K requests cost very little in total, may be the best option.

The metric to watch is cost per successfully completed task. Add SLA and you get cost per successfully completed task under SLA, which is much closer to real business value.

At that point, the gateway's job is no longer just to provide a unified endpoint. A more sensible architecture looks like this:

There are really four routing dimensions here: model, provider, service tier, and execution mode. The optimization space is far larger than simply picking "model A on provider B".

Closing: next-generation inference needs more than more models

For the past few years, the main question in AI infrastructure was: which model should I call? The industry answered with model gateways, model routers, provider routers, and fallback.

As agents and offline workloads grow, the question is becoming: how should this request be executed?

• Some requests need to be as fast as possible → Priority

• Some are normal production → Standard

• Some are not urgent but still have synchronous dependencies → Flex

• Some run so long that holding a connection is not worth it → Async

• Some are, by nature, a hundred thousand independent tasks → Batch

A mature inference platform will need to understand three things at once: urgency, execution, and scale. In other words: how urgent is this? Does the caller need to wait? Is this one task, or a dataset?

As model capabilities keep improving, the most valuable next step for AI infrastructure may not be pushing every request through the same API. It may be sending each type of workload down the execution path that suits it best.

Often, the most effective way to cut AI cost is not switching to a cheaper model. It is asking first: does this request really need to finish right now? And one step further: should it exist as a synchronous HTTP request at all?

Originally published in Chinese by Han (Andrew) Zheng on Substack.