The prompt cost optimizer above separates the two ways to spend less on input tokens and prices them individually: sending fewer tokens, and paying a reduced rate for the tokens you send repeatedly. They are frequently confused, they interact, and the order in which you apply them changes the result.
Arb Digital runs this comparison when a client's inference bill grows faster than their traffic, which is the usual pattern. Prompts accumulate instructions the way codebases accumulate comments — each addition is small, nothing is ever removed, and eighteen months later the system prompt is four times the size of the user's actual question and is re-sent on every request.
What This Prompt Cost Optimizer Does
Enter your current prompt length and a realistic trimmed length, then describe the cacheable portion and the hit rate you expect. The headline number is the total monthly saving. The grid shows the current bill, the optimised bill, and — the useful part — how much of the saving comes from trimming versus caching.
That split matters because the two require different work. Trimming is prompt engineering and evaluation. Caching is an integration change with almost no quality risk. If the calculator shows most of the saving coming from caching, you can capture it in an afternoon. If it comes from trimming, you have a testing exercise ahead of you.
How to Use It
- Measure your current prompt. Take a real production request, not a template — the prompt token estimator converts the exact text to a token count.
- Set a trimmed target. Be honest. Halving is realistic for a prompt that has never been reviewed; a 90% cut usually means you removed something that was doing work.
- Identify the cacheable prefix. This is the leading portion that is byte-identical on every request: system instructions, tool schemas, fixed examples. Anything after the first varying token cannot be cached.
- Estimate the hit rate. High-traffic endpoints with a stable prefix reach high rates easily; sporadic traffic that idles past the cache lifetime does not.
- Enter your prices and monthly volume, then click Calculate and read the two saving figures separately.
The Formula / How It's Calculated
Today's monthly cost is (current prompt × input rate + output × output rate) × volume ÷ 1,000,000. The trimmed cost substitutes the shorter prompt at the same rates, and the difference between them is the trimming saving.
Caching splits the trimmed prompt into a cacheable prefix and a variable remainder. On a hit you pay prefix × cache read rate + remainder × input rate. On a miss you pay prefix × cache write rate + remainder × input rate, with the write rate above the standard input rate. Blended by hit rate, the input cost per request becomes hit% × hit cost + (1 − hit%) × miss cost, and the caching saving is the trimmed cost minus that. Output cost is identical in all three scenarios, which is why it never appears in the saving.
Caching Has a Break-Even Hit Rate
Because a cache write costs more than an ordinary input token, caching is not free insurance. Below a certain hit rate you pay the write premium often enough to lose money. With a write premium of 25% and a read discount of 90%, the break-even hit rate sits at roughly a fifth — above that you save, below it you do not.
That threshold is easy to miss under sporadic traffic. Caches have a lifetime, often measured in minutes, and an endpoint receiving one request every few minutes may write the cache almost every time and read it almost never. Set the hit rate field low and watch the caching saving go negative. If your traffic is that thin, the answer is not caching — it is trimming, which helps at any volume.
Hit Rate Is Determined by Prompt Order
Caching matches a prefix. Everything from the first differing token onward is uncacheable, which means a single dynamic value near the top of a prompt destroys the cacheability of everything below it. A timestamp, a user name, a session identifier, or a randomly ordered set of retrieved chunks placed before the static instructions can take a prompt that should cache at 95% down to nothing.
The fix is structural and cheap: static first, dynamic last. System instructions, tool definitions, and fixed examples at the top, then retrieved context, then conversation history, then the user's message. Keep serialisation deterministic too — if your tool schemas are generated from a dictionary with unstable key order, the prefix differs between requests even though the content is identical. Provider documentation sets out exactly how prefixes are matched and how long cache entries live: see Anthropic's prompt caching documentation and OpenAI's prompt caching guide for the specifics of each.
What Actually Compresses in a Prompt
Some things shrink dramatically with no quality cost. Few-shot examples are usually the largest single opportunity: prompts frequently carry eight or ten demonstrations where three well-chosen ones perform identically, and each removed example is hundreds of tokens off every request forever. Repeated instructions are the second — the same rule stated three ways in three places, added by three different people, does not triple compliance.
Formatting is nearly free money. Minified JSON in examples, no decorative separators, no long preamble explaining the assistant's identity in prose when a sentence does. Retrieved context is the other big lever, though it belongs to retrieval tuning rather than prompt writing: reducing top-k from ten to five halves that portion, and the RAG chunk size calculator quantifies it. What does not compress safely is the output specification. Vague formatting instructions produce malformed responses, and a retry costs the entire prompt again — a saving that triggers a 5% failure rate is not a saving.
Measure Quality Before and After, Not Just Cost
Trimming is a change to model behaviour, and it should be treated like one. Build a set of representative inputs with known-good outputs before you cut anything, run it against the current prompt to establish a baseline, then re-run it after each removal. Without that, "the prompt still seems to work" is doing all the quality assurance in your pipeline.
Cut in stages and measure between them. Removing five things at once and finding a regression tells you nothing about which one caused it. Watch the edge cases in particular: a trimmed prompt often performs identically on typical inputs and falls apart on the unusual ones the removed instructions were written for in the first place. Once you have a stable optimised prompt, feed the numbers into the LLM cost comparison tool — a shorter prompt sometimes brings a cheaper model within reach, which saves more than either lever here.
Arb Digital audits prompt architecture, caching, and model selection together, and quantifies each change against a quality baseline before it ships.
Browse Free Tools Talk to Arb DigitalCommon Mistakes to Avoid
- Caching before trimming — a cached bloated prompt is cheap bloat, and you leave the larger saving untouched.
- Putting dynamic values early in the prompt — one changing token near the top makes everything after it uncacheable.
- Assuming caching always pays — write operations cost more than standard input, so low hit rates can increase the bill.
- Trimming without an evaluation set — quality regressions from removed instructions surface on edge cases, not on the examples you tested by hand.
- Cutting output format instructions — malformed responses trigger retries that cost the whole prompt again and wipe out the saving.
Related Free Tools From Arb Digital
Count what you are sending with the AI token calculator or the ChatGPT token counter, budget the full request in the context window planner, model agent loops with the AI agent cost calculator, price the result with the GPT API cost calculator, and compare providers in the LLM cost comparison tool. More in the free online tools hub.
Frequently Asked Questions
The provider stores the processed state of an identical leading portion of your prompt and charges a reduced rate to reuse it. Writing to the cache costs slightly more than a normal input token, and reading from it costs substantially less, so the saving depends on how often you hit.
It depends on the write premium and the read discount your provider applies. With a modest write premium and a large read discount, break-even typically sits at a low hit rate, but sporadic traffic that expires between requests can write far more often than it reads.
Usually because something dynamic appears early in the prompt. Caching matches a prefix, so a timestamp, session identifier, or non-deterministic serialisation near the top makes everything after it uncacheable.
Trim first. Caching reduces the rate you pay on repeated tokens but does not remove them, so optimising the prompt before caching means you are paying a discounted rate on a smaller amount of text.
Redundant restatements of the same rule, surplus few-shot examples beyond the few that establish the pattern, decorative formatting, and verbose persona description. Output format specifications should be left alone, since vague instructions cause failures that trigger costly retries.
It generally does, because the provider skips reprocessing the cached prefix. That reduces time to first token for applications with a large fixed prompt, which is often as valuable as the cost saving.
Not directly. Shortening the prompt reduces input tokens only. Output cost changes only if the shorter prompt changes how long the model's responses are, which is worth checking during evaluation.
Figures produced by this tool are planning estimates only — actual savings depend on your provider's current rates, cache lifetime and matching rules, and how your traffic is distributed over time.
