The inference batching calculator above models the central trade-off in serving a language model on your own hardware: processing several requests together makes each one much cheaper and every one of them slower. It reports cost per thousand requests, the throughput your configuration sustains, and the latency an individual user experiences, so you can find the batch size that satisfies both a budget and a service level rather than optimising one and discovering the other later.
Arb Digital uses this when a client is deciding between a hosted API and self-hosted inference, because self-hosting only wins on cost if the hardware stays busy. An expensive accelerator running one request at a time is the most costly way to serve a model, and batching is the mechanism that fixes it.
What This Inference Batching Calculator Does
Enter your batch size, output length, single-request generation speed, the batch scaling exponent for your setup, per-batch overhead, arrival rate, and hourly compute cost. The headline result is cost per thousand requests. The grid shows sustained throughput, end-to-end latency per request, the portion of that latency spent waiting for the batch to fill, and how much cheaper the configuration is than serving requests one at a time.
The bars decompose latency into queueing, decoding, and fixed overhead. Which segment dominates tells you what to change: a large queue bar means the batch is too big for your arrival rate, while a large decode bar means the batch is saturating your hardware and you need either more of it or a smaller model.
How to Use It
- Set batch size and output length. Use a realistic average output; the AI latency estimator helps if you are still characterising responses.
- Measure single-request speed. Time one request with no other load and divide output tokens by generation seconds.
- Set the scaling exponent. Measure it if you can: run at batch 1 and batch 8, then compare aggregate token throughput. Values around 0.6 to 0.8 are common for memory-bound decoding.
- Enter arrival rate and hourly compute cost. Arrival rate drives queue wait, which is often the larger half of latency at big batch sizes.
- Step the batch size up — 1, 2, 4, 8, 16, 32 — and watch where cost stops improving while latency keeps rising.
The Formula / How It's Calculated
Aggregate throughput is modelled as single-request speed × batch size ^ exponent. With an exponent of 0.7, a batch of 16 gives about seven times the token throughput of a single request, not sixteen times. A batch then takes (batch × output tokens) ÷ aggregate throughput + overhead seconds to complete, and served throughput is batch ÷ that time requests per second.
Average queue wait is approximated as batch ÷ (2 × arrival rate), since a request arriving at a random point waits on average for half the batch to assemble. Total latency is queue wait plus batch time. Cost per thousand requests is hourly cost ÷ (throughput × 3,600) × 1,000. If served throughput falls below arrival rate, the queue grows without bound and the tool says so — that configuration has no steady state, whatever its cost per request looks like.
Why Throughput Scales Sublinearly
Batching helps because token generation on a modern accelerator is limited by memory bandwidth rather than arithmetic. Producing one token requires reading the model's weights out of memory, and that read costs the same whether one sequence or thirty-two are waiting for it. Batching amortises the read across every sequence in the batch, which is why the first few doublings are close to free.
It stops being free because each sequence in the batch carries its own key-value cache, and those grow with sequence length. Past a point the batch is competing for the same memory bandwidth that made batching worthwhile, and the returns flatten. That is what the exponent captures. Rather than trusting the 0.7 default, measure your own: it varies with model size, quantisation, sequence length, and hardware, and it is the input this calculation is most sensitive to.
Latency Is Linear While Savings Are Not
This is the shape of the trade-off and the reason the tool exists. Cost per request improves steeply from batch 1 to batch 8, noticeably from 8 to 16, and marginally beyond 32. Latency does the opposite: queue wait rises in direct proportion to batch size, and it never flattens.
Step through the batch sizes and you will usually find a knee — a point past which you are paying real latency for a saving measured in fractions of a cent. For an interactive assistant that knee is the answer. For an overnight document processing job there is no knee, because nobody is waiting, and you should push the batch as high as memory allows. The right batch size is a property of the workload, not of the hardware, which is why a single global setting for a service handling both interactive and background traffic is nearly always wrong for one of them.
Continuous Batching Changes the Shape
The static model above — assemble a batch, run it to completion, return everything — is what naive implementations do, and it wastes a great deal of capacity. Sequences in a batch finish at different times, and under static batching the short ones sit idle waiting for the longest to complete.
Continuous or in-flight batching removes that waste by evicting finished sequences and admitting new ones every step, keeping the batch full without forcing anyone to wait for a fixed group. Modern serving stacks implement this by default; NVIDIA's Triton model configuration documentation covers dynamic batching and the queue delay settings that control it. If you are using such a stack, treat this calculator's queue wait as a conservative upper bound: continuous batching typically cuts it substantially while leaving the cost side broadly intact.
Offline Batch APIs Are a Different Product Entirely
If you use a hosted API rather than your own hardware, "batching" usually means an asynchronous endpoint: you submit a large job, the provider schedules it against spare capacity, and results arrive within a stated window rather than in seconds. The discount for accepting that delay is significant, and it applies to work where nobody is waiting — evaluation runs, backfills, bulk classification, content enrichment.
The economics are different from self-hosted batching. There is no hardware to keep busy and no batch size to tune; you are simply trading latency for a lower rate. OpenAI's Batch API guide documents how jobs are submitted and what turnaround to expect. Model the saving by entering the discounted rate into the LLM cost comparison tool, and check whether your synchronous traffic would then fit inside your existing limits using the LLM throughput calculator.
What Caps Your Batch Size in Practice
Memory, not throughput. Every active sequence holds a key-value cache proportional to its context length, so a batch of 32 requests with 8,000-token prompts can require many times the memory of the same batch with 500-token prompts. Exceeding available memory does not degrade gracefully — the server fails or evicts sequences mid-generation.
This makes prompt length a capacity constraint as well as a cost one. Shortening prompts increases the batch size your hardware can hold, which improves throughput and cost per request simultaneously; the prompt cost optimizer quantifies that side. It also argues for segmenting traffic: long-context requests and short ones behave so differently under batching that routing them to separate pools, each tuned to its own shape, usually beats a single compromise configuration serving both badly.
Arb Digital models utilisation, batching, and latency targets together, so the comparison reflects the load you will actually serve rather than a benchmark.
Browse Free Tools Talk to Arb DigitalCommon Mistakes to Avoid
- Assuming throughput scales linearly with batch size — decoding is memory-bound, so returns flatten well before the batch is large.
- Using one batch size for interactive and background traffic — they have opposite latency requirements and belong in separate pools.
- Ignoring queue wait — at large batch sizes and low arrival rates, most of a user's wait is spent assembling the batch, not generating tokens.
- Sizing the batch from throughput alone — the real ceiling is key-value cache memory, which scales with context length as well as batch size.
- Comparing self-hosted cost at full utilisation — idle hardware is billed the same, so cost per request depends on the load you actually receive.
Related Free Tools From Arb Digital
Estimate response time in the AI latency estimator, check capacity against limits with the LLM throughput calculator, price hardware in the GPU cost calculator, compare against hosted rates in the LLM cost comparison tool, and size prompts with the AI token calculator. More in the free online tools hub.
Frequently Asked Questions
Generating a token requires reading the model weights from memory, and that read is shared across every sequence in the batch. Spreading a fixed memory cost over more requests lowers the compute time each one consumes, and therefore the cost.
Because decoding is limited by memory bandwidth, and each sequence carries its own key-value cache that competes for the same resource. Aggregate throughput rises with batch size but at a decreasing rate, which the scaling exponent represents.
The one where cost per request stops improving meaningfully while latency is still acceptable. That point differs for interactive and offline work, so a service handling both usually needs separate pools rather than a single compromise value.
It is a scheduling approach that admits new sequences and evicts finished ones on every generation step, instead of running a fixed group to completion. It keeps hardware busy and reduces queue wait compared with static batching.
Available memory for the key-value cache, which scales with both batch size and context length. Long prompts reduce the number of sequences that fit, so shortening prompts raises the batch size the same hardware can serve.
No. A hosted batch endpoint is an asynchronous job service that trades turnaround time for a lower rate. There is no batch size to tune, and the saving comes from the provider scheduling your work against spare capacity.
Because served throughput is below the arrival rate. No batch size fixes that on its own — the configuration needs more hardware, shorter outputs, or a smaller model before it has a stable steady state.
Figures produced by this tool are planning estimates only — actual throughput and cost depend on your model, hardware, serving stack, sequence lengths, and how evenly traffic arrives.
