AI Cost in Production Is Set by Prompt Design, Not Model Choice: A Prompt Caching Guide for Engineers

When a company's AI bill climbs, the first instinct is usually the same: find a cheaper model. In production, though, the biggest cost driver is often not the model but how the prompt is assembled. An agent that works through 40 turns resends the entire conversation on every turn, so the cost of a single task grows roughly with the square of the number of turns. Prompt caching does not stop that resending, but it sharply cuts the price of everything that has already been sent. In measurements Anthropic has published, caching cut agent-loop cost by a factor of 2.5 to 3.7 at cache hit rates of 81 to 90 percent.

This article is written for engineers running AI in production, with a summary up front so decision makers can use it too. The numbers and behavior below refer to Anthropic's API. Other providers use different schemes (some cache automatically with no markers, some bill storage by the hour), so check their documentation before copying these assumptions.

Summary for decision makers

• Reading from the cache is priced at about 10 percent of the normal input price. Writing to it carries a one-time premium: 1.25 times (5-minute lifetime) or 2 times (1-hour lifetime).
• Two or three requests sharing the same prefix are enough to break even.
• The most expensive failure is the silent one: requests keep succeeding and only the bill rises. The evidence lives in the usage numbers, not in the code.
• The order of cost levers: free ones first (caching, batch, input hygiene), then the ones that trade quality (effort, model).

One rule decides everything: prefix matching

The cache matches on the exact bytes of the rendered prompt, from the start up to a cache_control breakpoint. A single byte that differs at position N invalidates everything after it. The render order is always the same: tool definitions, then the system prompt, then messages. The consequence is simple: the most stable content goes first and the most volatile goes last.

If a timestamp is interpolated into the system prompt header, nothing after it can be cached, no matter how many breakpoints you add. The four placement patterns used most often:

• A large system prompt shared across many requests: put the breakpoint on the last system block. Tools and system are cached together.
• Multi-turn conversations: put the breakpoint on the last block of the newest turn, or turn on automatic caching at the request level. Every turn reads the full prior history.
• Same prefix, varying suffix: put the breakpoint at the end of the shared portion, not the end of the whole prompt. Otherwise every request writes a new entry and nothing is ever read.
• A prompt that changes from the very first byte: do not cache it. You would only pay the write premium and never read.

The limit is four breakpoints per request. For long-running agents, a strong combination is one explicit breakpoint at the end of the static system prompt plus automatic caching for the growing conversation tail.

The numbers worth memorizing

Break-even comes from total cost. With a 5-minute lifetime, two requests are enough: 1.25x plus 0.1x is 1.35x, against 2x without a cache. With a 1-hour lifetime you need three: 2x plus 0.2x is 2.2x, against 3x. On the newest top-tier model the read price drops further, to US$0.25 per million tokens, which is 0.025 times the normal input price, and every break-even calculation shifts with it.

Three details are often missed:

• The minimum prefix size differs by model. The newest models need only 512 tokens, but some earlier-generation models require up to 4,096. The number does not rise or fall in an orderly way across generations. A 3,000-token prompt can be cached on one model and silently not on another, with no error at all, only a cache_creation_input_tokens value of zero.
• Reading the cache refreshes its timer at no extra cost. The lifetime is measured from the start of a request, not its end, so generation time eats into the budget.
• Caches are isolated per workspace. Traffic for the same prompt that is split across workspaces writes and reads separate entries. Check this before blaming the prompt for a low hit rate.

Pick the lifetime from the gap between requests, not from instinct

Measure the gap from the start of one request to the start of the next that shares the prefix. If the gap is always under 5 minutes, the 5-minute lifetime refreshes itself on every request and it is the cheaper choice. The 1-hour lifetime only pays off in the 5 to 60 minute window, such as a user who replies 20 minutes later or a long side task. Beyond an hour, neither helps: re-warm the cache on a schedule or accept the first miss.

For the newest top-tier model with its 0.025x read price, there is an option that is usually cheaper than the 1-hour lifetime: stay on the 5-minute lifetime and, while idle, resend the last request with max_tokens: 0 just before the entry expires. That request refreshes the timer and bills only a very small read charge, with no output tokens.

There is one concurrency trap. A new cache entry becomes readable only after the first response begins streaming. N parallel requests with an identical prefix all pay full price. For a fan-out pattern, send one request first, wait for its first token, then launch the rest.

Five ways a cache dies silently

All of these share one trait: requests keep succeeding. That is why no alarm rings until someone opens the bill.

Two more things deserve attention. First, each breakpoint looks back only 20 positions to find a prior entry. A single turn that adds more than 20 positions of content (long chains of sequential tool calls, many images) can push the previous entry out of the window, and the whole conversation is rewritten on every request even though the payloads are byte-identical. The fix: place an intermediate breakpoint roughly every 15 positions. Second, changing the thinking or effort configuration invalidates the message cache, and on some models the tool and system caches too. Pin both per route instead of varying them per request.

Verify from the usage numbers, not from the code

Three fields on usage are the only proof that caching works: cache_creation_input_tokens (tokens written, at the premium), cache_read_input_tokens (tokens read from the cache), and input_tokens (the remainder, paid in full). The sum of the three is the real size of the prompt. Looking at input_tokens alone is misleading: an agent that ran for hours can appear to process only 4 thousand tokens because the rest was served from the cache.

A healthy loop has a clear signature. On every request, cache_read_input_tokens covers the entire prior history and keeps growing, cache_creation_input_tokens is roughly the size of the last output plus the newly appended input, and input_tokens is only the tail after the last breakpoint. If the write count is close to the size of the whole conversation on every request, something upstream is breaking the prefix.

The costliest production failure is almost always a regression, not a flawed first implementation. Caching works when first built, then a prompt-assembly change a month later (a new dynamic field, a feature that rewrites history, a tool list whose order is no longer fixed) makes every request miss and nobody notices for months. So install a standing check: an integration test that sends an identical request twice and asserts that cache_read_input_tokens is above zero on the second, or monitoring on all three fields.

To trace a cause, compare the payloads of consecutive requests after stripping the cache_control markers (the moving marker always differs and is not a cause). The first divergence inside the region that should be identical is the invalidation point. Anthropic also offers a cache diagnostics feature (beta) that names where two requests diverged without any payload logging.

What we learned from our internal agents

We run AI agents for internal work, and two decisions proved more decisive for cost than the choice of model.

Keep small jobs to a single model call with no tools. For one-shot reasoning tasks, such as analyzing one inbound message and suggesting an action, we use a single bounded call: a short system prompt, no tool access, no file or shell access. The recorded cost per call is roughly 1 to 5 US cents, and the behavior is easy to predict. There is a detail that bites here. Tools are turned off with an empty tool list, not an empty allow list, because the latter only controls automatic approval and the tools remain available. We also once capped the turn count at one and it failed, because the number of turns the SDK uses varies in undocumented ways. A cap of five turns solved it.

Align the session freshness window with cache behavior. Conversations with an agent can be resumed. We resume a session that was active within the last hour and start a new one after that. The reasoning is cache economics: while the cache is warm, resuming a long history is nearly free. Once it has gone cold, that long history is paid in full on the first turn, so a fresh session is cheaper. A manual control to start a new session is always available.

The order of cost levers: free ones first

AI cost should be optimized in units of cost per completed task, not cost per token or per request. A model that costs more per token can be the cheaper one if it finishes the job in fewer turns. A cheap model that fails still bills its tokens, then the retry, then whatever the failure costs downstream.

The free levers go first because they do not lower output quality:

• Caching: the largest lever on every model and benchmark Anthropic measured. Once it is on, the job is keeping it healthy.
• Input hygiene: move large reference documents behind a tool so they are fetched on demand, remove tool recaps that merely repeat the schema, and downscale images. Images are counted by pixel area, roughly one token per 28 by 28 pixel patch, so 1280 by 720 stops at about 1,200 tokens.
• Output and loop: max_tokens is a backstop, not a lever. The model never sees it, and a truncated response is a failed attempt. Control length through the output shape requested in the prompt, with an example.
• Batch: a 50 percent discount on every token, including cache reads and writes, for work nobody is waiting on. Results arrive within 24 hours.

Only after that come the levers that trade quality, one at a time and measured against an eval: lower effort first, then consider the model. On research and knowledge workloads the curve is nearly flat. On long-horizon coding there is a real tradeoff. And remember that prompts written for an older model can make a newer one over-work: in one Anthropic evaluation on customer support, prompts written for the previous generation cost 36 percent more per ticket with no change in accuracy.

Closing

Healthy AI cost comes from prompt-assembly discipline and a habit of verifying, not from luck. Engineers who put stable content first, keep the prefix deterministic, and treat the usage numbers as an alarm will pay far less for the same quality, on any model.

If your team runs AI agents in production and wants to review prompt assembly, the cache verification path, or the cost architecture, we are open to an initial conversation.