The AI latency estimator above breaks an LLM response into the parts that actually determine how slow it feels: the network round trip, the prefill pass over your prompt, and the token-by-token generation of the answer. It reports the total elapsed time and, separately, the perceived wait when the response streams — two numbers that can differ by an order of magnitude for the same request.
Arb Digital uses this model when specifying AI features, because latency targets are usually written as a single number ("under three seconds") without any view of which component is responsible. Once the request is broken into its parts, it is normally obvious which one to attack, and it is rarely the one the team assumed.
What This AI Latency Estimator Does
Enter prompt and output length, the prefill and generation throughput you expect from your model and hardware, and your network overhead. The headline result is total wall-clock time from request to final token. The grid separates time to first token, generation time, how far the model outpaces a reader, and how long a human takes to read the answer — the last two being the reference points that tell you whether further optimisation is worth anything.
The bars show where the milliseconds go. On short prompts the network dominates. On long-context requests prefill takes over. On long answers generation is the whole story. Each of those has a different fix, and applying the wrong one is a common way to spend a sprint for no measurable improvement.
How to Use It
- Enter prompt tokens. Use the full assembled prompt including system instructions, history, and retrieved context — the context window planner gives you that total if you have not added it up.
- Enter expected output tokens. Estimate from real answers, not from your maximum-tokens setting, which is a ceiling rather than a prediction.
- Set prefill and generation speed. If you do not know them, time a real request: divide output tokens by the seconds spent generating to get tokens per second.
- Set network and queue overhead. Measure it as the gap between sending a request and the first byte on a trivial prompt, which isolates it from compute.
- Click Calculate and judge the result by time to first token and reading time rather than by the total.
The Formula / How It's Calculated
Two phases, priced in time rather than money. Prefill processes the entire prompt in one parallel pass: prompt tokens ÷ prefill speed. Decoding then produces the answer one token at a time, each token conditioned on all the ones before it: output tokens ÷ generation speed. Time to first token is network overhead plus prefill; total time is that plus generation.
At the defaults, 4,000 prompt tokens at 3,000 tokens per second is about 1.3 seconds of prefill, plus 0.25 seconds of network, giving roughly 1.6 seconds to first token. Then 500 output tokens at 60 per second adds another 8.3 seconds, for about 10 seconds total. Reading 500 tokens — roughly 385 words — at 240 words per minute takes about 96 seconds. The user is reading far slower than the model is writing, which is why streaming makes a ten-second response feel like a 1.6-second one.
Time to First Token Is a Function of Prompt Length
Prefill is parallel, so it is fast per token, but it is not free and it scales with prompt size. Doubling a prompt roughly doubles prefill time. This is the mechanism behind a complaint teams hear constantly: the assistant "got slower" after retrieval was added, or after the system prompt grew, even though the answers are the same length.
It also explains why prompt caching improves latency and not just cost. When a provider can reuse the computed state of a prompt prefix, the prefill work for that prefix is skipped, and time to first token drops accordingly. For a chat application with a large fixed system prompt, this is often the single largest available latency win, and it requires no model change. Model the cost side of the same change in the prompt cost optimizer.
Streaming Changes the Experience, Not the Clock
Streaming does not make generation faster. It changes when the user first sees something from the total elapsed time to the time to first token. Research on interface response times — the thresholds documented in Nielsen Norman Group's work on response time limits — puts roughly one second as the boundary for uninterrupted flow of thought and about ten seconds as the limit of a user's attention on a task. A non-streamed ten-second answer sits right at the edge of abandonment. The same answer streamed, with the first words appearing in under two seconds, stays comfortably inside the flow threshold.
The corollary is that once generation outpaces reading, further throughput gains are invisible to the user. If your model produces 60 tokens per second and readers consume the equivalent of about five, tripling generation speed changes nothing perceptible. That effort belongs on time to first token instead, or on throughput if your constraint is concurrent users rather than individual experience — the LLM throughput calculator covers that case.
Where Production Latency Actually Comes From
The clean two-phase model above describes an unloaded system. Real deployments add several sources of delay that no token count predicts. Queueing is the biggest: when a provider or your own inference server is saturated, requests wait before any compute starts, and that wait grows non-linearly as utilisation approaches capacity. Rate limit retries with exponential backoff can add seconds that are invisible in your application logs if the client library handles them silently.
Multi-step systems multiply everything. An agent that makes six model calls does not take six times one call's prefill — it takes progressively longer prefills as the conversation grows, plus the execution time of each tool between calls. Measure end-to-end and at the ninetieth and ninety-ninth percentiles, because averages hide exactly the requests that cause users to give up. The AI agent cost calculator shows the same accumulation problem on the cost axis.
Structured Output and Tool Calls Undo the Streaming Advantage
Streaming only helps when partial output is useful to the reader. When a model returns JSON that your application must parse before rendering anything, the user sees nothing until the last token arrives, so the perceived wait equals the total time regardless of whether streaming is enabled. The same applies to tool calls: the arguments have to be complete before the tool can run.
Two practical responses. Design output so the human-readable part comes first and the structured part last, letting you stream the prose while the machine-readable payload is still forming. And where a step genuinely cannot stream, show the specific work in progress rather than a generic spinner — "searching your documents" reads as progress in a way that an indeterminate loader does not. OpenAI's streaming guide documents how partial responses are delivered and what arrives in what order.
Arb Digital diagnoses where the seconds actually go — prompt size, queueing, or generation — and rebuilds the request path around the number users feel.
Browse Free Tools Talk to Arb DigitalCommon Mistakes to Avoid
- Optimising total time when the interface streams — users judge the wait by the first token, so that is the number to reduce.
- Ignoring prompt length as a latency factor — prefill scales with prompt size, so a longer system prompt slows every response before generation begins.
- Measuring only averages — the ninety-ninth percentile is where queueing, retries, and unusually long answers live, and it is what drives abandonment.
- Assuming streaming always helps — output that must be parsed before rendering shows the user nothing until it is complete.
- Chasing generation speed past reading speed — once the model outpaces the reader, extra throughput is invisible in a single-user experience.
Related Free Tools From Arb Digital
Measure your prompt with the prompt token estimator, budget the whole request in the context window planner, check concurrency limits with the LLM throughput calculator, weigh batch trade-offs in the inference batching calculator, and compare model characteristics in the AI model comparison tool. More in the free online tools hub.
Frequently Asked Questions
It is the delay between sending a request and receiving the first piece of the answer. It consists of network and queue overhead plus the prefill pass over your prompt, so it grows with prompt length and is independent of how long the answer will be.
The model must process the entire prompt before it can generate anything. That prefill pass is fast per token because it runs in parallel, but it still scales with prompt size, so adding retrieved context or a longer system prompt delays the first token.
No. It delivers tokens as they are produced instead of waiting for the complete answer, so the user starts reading much sooner. Total generation time is unchanged; the perceived wait drops to time to first token.
It varies widely by model size, hardware, quantisation, and how loaded the service is. Rather than relying on a published figure, time a representative request of your own and divide output tokens by the seconds spent generating.
Prefill processes all prompt tokens in parallel in a single pass. Generation is sequential by nature, because each new token depends on the tokens produced before it, so it cannot be parallelised in the same way.
It can, because reusing the computed state of a repeated prompt prefix skips the prefill work for that portion. For applications with a large fixed system prompt, that often produces a noticeable reduction in time to first token.
This model assumes an unloaded system. Real deployments add queueing under load, retries after rate limit errors, tool execution time, and network variance, all of which show up most strongly in high-percentile measurements rather than averages.
Figures produced by this tool are planning estimates only — actual latency depends on model, hardware, provider load, network conditions, and how your application assembles each request.
