The LLM throughput calculator above takes your peak traffic and your provider's rate limits and answers three questions: which limit you hit first, how much room is left before throttling starts, and how many requests you need in flight at once to sustain the load. Rate limits are usually expressed as two separate ceilings, and applications almost never hit both at the same time — knowing which one binds decides whether you shorten prompts, request a quota increase, or add a second deployment.
Arb Digital runs this check before an AI feature launches, because rate limit failures do not appear in staging. They appear at the traffic peak, on the day the feature is promoted, and they look to users like the product is broken rather than busy.
What This LLM Throughput Calculator Does
Enter your peak requests per second, the token shape of an average request, your requests-per-minute and tokens-per-minute limits, and typical request duration. The headline figure is the requests per second your limits actually sustain. The grid shows how much of each limit your traffic consumes as a percentage, the concurrency required to carry it, and the headroom left before throttling begins.
The two bars make the binding constraint obvious at a glance. If the token bar is at 95% while the request bar sits at 30%, adding request quota changes nothing — you need shorter prompts, a larger token allowance, or traffic spread across more than one deployment.
How to Use It
- Enter peak requests per second. Take your busiest minute from application logs and divide by 60. Averages hide exactly the moment limits are enforced.
- Enter input and output tokens per request. Include everything you send — system prompt, tools, history, retrieved context. The context window planner totals it for you.
- Enter your real limits. These are specific to your account, model, and usage tier, and they change as your account matures.
- Enter average request duration, measured end to end. The AI latency estimator will produce a figure if you do not have production data yet.
- Click Calculate and read the bars first — the binding limit determines which fix is worth pursuing.
The Formula / How It's Calculated
Your traffic converts to per-minute terms as requests per second × 60 for the request limit, and requests per second × 60 × (input + output tokens) for the token limit. Utilisation is each figure divided by the corresponding ceiling. The maximum sustainable rate is the lower of RPM ÷ 60 and TPM ÷ 60 ÷ tokens per request, because both must be satisfied at once.
Required concurrency comes from Little's Law, the queueing result formalised in John Little's classic proof: the average number of items in a system equals arrival rate multiplied by time in the system. Here that is requests per second × average duration. Six requests per second, each taking eight seconds, means 48 requests in flight at all times. That number, not the request rate, sizes your connection pool, your worker count, and your timeout policy.
The Token Limit Binds First, Almost Always
Try the defaults, then double the input tokens to simulate adding retrieval. The request bar does not move. The token bar doubles. That asymmetry is the normal case for any application with a substantial prompt: RPM constrains high-frequency, short-prompt workloads like classification, while TPM constrains everything with long context.
This matters because the instinctive response to throttling is to ask for more requests per minute, which for most applications is quota that will never be used. The effective moves against a token ceiling are different: trim what you send on every request, cache the repeated prefix, or move some traffic to a different model or region with its own allowance. Provider documentation is explicit about how the two ceilings interact — OpenAI's rate limits guide and Anthropic's rate limits reference both describe the separate request and token buckets and how they refill.
Output Tokens Are Counted Before They Exist
This is the detail that turns a comfortable calculation into unexplained throttling. Providers generally cannot know how long a response will be at the moment the request arrives, so they reserve capacity against your token limit using the maximum output you requested. Set a maximum of 4,000 tokens on a request that typically returns 300, and you may be charged 4,000 against your per-minute allowance for admission purposes even though you are billed for 300.
At scale that difference is the entire margin. An application setting a generous maximum "just in case" can consume many times its actual token throughput in limit terms and be throttled at a fraction of its apparent capacity. The fix costs nothing: set the maximum output close to what your responses genuinely need, per endpoint rather than globally. Enter your configured maximum in the output field above, not your observed average, to see the ceiling your limits are really enforcing.
Per-Minute Limits, Per-Second Traffic
Limits are quoted per minute and typically enforced with a continuously refilling bucket rather than a counter that resets on the minute. A workload that fires its entire minute's worth of requests in the first five seconds can be throttled even though its minute total sits well inside the quota, because the bucket empties faster than it refills.
Batch jobs are the usual offender. A nightly process that iterates a list as fast as its worker pool allows will hit limits immediately, retry, and produce a sawtooth of failures. Adding a client-side rate limiter that paces requests at the sustainable rate this calculator reports is the correct fix, and it usually finishes the job faster than an unpaced loop fighting rejections. Real user traffic is bursty too — comfortable average utilisation with three-times peaks means you are designing for a limit you exceed several times a day.
Designing for the Throttle You Will Eventually Hit
Assume rejection will happen and decide in advance what the application does. Exponential backoff with jitter is the baseline: retry after a growing delay, randomised so that every throttled client does not retry in unison and recreate the spike. Respect any retry-after header the provider returns rather than guessing, and cap total retry time so a user is not waiting behind a queue that will not clear.
Beyond retries, the useful patterns are queueing and degradation. Work that does not need an immediate answer — summarisation, enrichment, classification — belongs in a queue that drains at your sustainable rate, leaving the synchronous allowance for user-facing requests. Where an answer is needed now, failing over to a smaller model with its own separate limit keeps the feature alive at reduced quality, which users tolerate far better than an error. Compare candidates for that fallback in the AI model comparison tool, and price the difference in the LLM cost comparison tool.
Arb Digital load-models AI features against real rate limits before launch, so the first traffic peak is a capacity plan rather than an incident.
Browse Free Tools Talk to Arb DigitalCommon Mistakes to Avoid
- Planning against average traffic — limits are enforced against your busiest minute, and peaks are commonly several times the mean.
- Requesting more requests per minute when tokens are the constraint — check which bar is higher before asking for a quota change.
- Setting a large maximum output token value by default — reserved capacity is counted against your token limit even when the response is short.
- Retrying without jitter — synchronised retries from every client rebuild the spike that caused the throttling.
- Ignoring concurrency — arrival rate times duration determines how many requests are in flight, and that is what sizes pools and timeouts.
Related Free Tools From Arb Digital
Measure request size with the prompt token estimator, confirm it fits with the token limit checker, model response time in the AI latency estimator, weigh queued work in the inference batching calculator, and cut what you send with the prompt cost optimizer. More in the free online tools hub.
Frequently Asked Questions
RPM caps how many requests you may send per minute regardless of size, while TPM caps the total tokens those requests may consume. Both are enforced at once, so your effective throughput is whichever ceiling you reach first.
For applications with long prompts — anything using retrieval, long system instructions, or conversation history — the token limit is normally reached long before the request limit. Short, high-frequency calls such as classification are the opposite case.
Yes, and providers commonly reserve capacity based on the maximum output you request rather than what the response actually uses. Setting a large maximum by default can consume far more of your allowance than your real usage.
Multiply arrival rate by average request duration. Six requests per second at eight seconds each means about 48 in flight at any moment, which is what sizes your worker pool and connection limits.
Limits are usually enforced with a continuously refilling bucket rather than a counter reset each minute. Sending a minute's worth of traffic in a few seconds empties the bucket faster than it refills, so requests are rejected despite an acceptable total.
Retry with exponential backoff and randomised jitter so clients do not retry in unison, respect any retry-after header the provider sends, and cap the total retry window. Non-urgent work belongs in a queue paced at your sustainable rate.
Most providers increase limits as account usage and history grow, and many accept requests for higher quotas. Before asking, confirm which ceiling actually binds, since extra capacity on the wrong limit changes nothing.
Figures produced by this tool are planning estimates only — actual limits depend on your provider, model, account tier, and how each provider accounts for reserved output tokens.
