A large language model retains no memory between API calls. Each request arrives as a blank slate, the model processing every input token from the beginning. In a ten-turn conversation, the model reprocesses the system prompt and all nine prior turns before generating the tenth response. In an agentic workflow that issues thirty tool calls, it performs this reprocessing thirty times.
Prompt caching eliminates this redundancy by storing the computational state from a request’s prefix and reusing it when the next request shares the same starting tokens. As agent workflows have grown heavier, this optimization has moved from a useful cost reduction to the infrastructure layer on which agentic economics depend.
Stored progress
When a request reaches an inference server, the model processes every input token, from system prompt and conversation history to the user’s message, and computes two vectors at each layer for each position: a key and a value. These KV tensors form the model’s working memory for the duration of the response, referenced during generation to decide which earlier tokens shape the output.
Normally, the server discards these tensors the moment the response finishes and rebuilds them from scratch on the next request. Prompt caching stores the KV tensors for the prefix of each request and checks incoming requests against this stored state before starting the forward pass. When the incoming prefix matches a stored entry, the server loads the cached tensors and runs computation only over the new tokens at the end, skipping the prefill phase for the entire matched prefix.
The mechanism depends on exact prefix matching. Tokens must align from the first position forward, byte for byte. A single changed token invalidates every cached tensor after it, because each position’s KV computation depends on all preceding positions through the attention mechanism. This constraint turns prompt structure into an engineering decision. The order in which content appears determines whether the cache can serve the request.
Agentic habits
A system prompt in 2024 might have carried a few hundred tokens of behavioral instructions. A system prompt in 2026 can carry tens of thousands, loaded with MCP tool definitions, conversation policies, and structured context that the model needs for every interaction. The protocol between client and server carries no memory of prior requests, so each call transmits this entire prefix as though the model has never seen the conversation.
Agentic workflows amplify the repetition. A single task, whether booking a flight, filing a report, or debugging a function, may generate dozens of sequential API calls as the agent invokes tools, reads results, and decides on next steps. Each of those calls carries the full system prompt plus the accumulated conversation history, with only the newest tool result or assistant message appended at the end.
In April 2026, Requesty measured that 77% of input tokens sent to Claude through Anthropic’s API came from cache. Anthropic’s servers had already processed three-quarters of those tokens on a prior request in the same session. Without prompt caching, every one of those tokens would have consumed a full forward pass.
Levers of control
Anthropic’s implementation requires developers to place explicit cache_control breakpoints in the API request, marking the boundary between the stable prefix and the dynamic content that follows. Cache reads cost one-tenth of the standard input rate. The initial write that populates the cache carries a small premium, but the economics break even after two reads of the same prefix. In an agentic workflow, that threshold passes within seconds.
OpenAI launched automatic prompt caching in late 2024 with no developer intervention. The system detected repeated prefixes and cached them transparently. With GPT-6 in September, OpenAI shifted toward explicit controls, adding cache breakpoints, a dashboard with miss diagnostics, and a prewarming mechanism that lets developers populate the cache before traffic arrives. The convergence suggests that automatic detection alone leaves performance on the table, and that production workloads benefit from deliberate prompt architecture.
Self-hosted inference engines solve the same problem at a lower layer of the stack. SGLang’s RadixAttention organizes all cached KV state in a radix tree, a data structure built for matching shared prefixes across requests. When a request arrives, the scheduler walks the tree to find the longest matching prefix and processes only the unmatched tokens, achieving up to five times the throughput of conventional serving on agentic workloads with shared system prompts.
All three approaches enforce the same prompt-design discipline. Stable content, including system instructions, tool definitions, and few-shot examples, must precede variable content. The boundary between the two determines the cache hit rate, and the cache hit rate determines the inference bill. Prompt architecture has become infrastructure architecture.
Optimized architecture
Reducing inference cost used to mean selecting a cheaper model or negotiating volume pricing. Prompt caching introduced a third lever that depends entirely on how the application structures its requests. Teams that organize their prompts for cache efficiency can cut input costs by 90% on the same model, at the same tier, from the same provider.
Every major provider now competes on cache tooling, from breakpoints and dashboards to prewarming and tiered pricing, because caching infrastructure determines who captures the agentic workload. As agent loops grow longer and system prompts grow heavier, the cost gap between a cache-aware application and one that ignores prompt structure will widen with every request.