Prompt caching is a feature of large language model (LLM) APIs that saves the processed form of the beginning of a prompt. When a later request starts with exactly the same text, the provider reuses that work instead of processing it again. Cached input is billed at a fraction of the normal price and the response starts sooner.
How It Works
Before a model writes its first output token it processes every input token. For a long system prompt, a set of tool definitions or a large reference document, that processing is a large share of the cost and the wait. A cache stores the result for a prefix, the exact sequence of tokens from the start of the prompt up to some point. A new request can reuse it only if its prefix is identical, character for character. One changed word near the top invalidates everything after it.
Providers implement this differently, and the details change often, so always check the current documentation:
- Anthropic. Developers mark a breakpoint with a cache control setting, or enable automatic placement. The cache covers tools, then the system prompt, then messages, in that order. The default lifetime is five minutes, refreshed on each hit, with a one-hour option. Writing to the cache costs more than normal input, and reading costs a small fraction of it (Anthropic prompt caching).
- OpenAI. Caching is on by default for supported models, applies above a minimum prompt length, and requires the whole rendered prefix to match. The documentation describes cached input as discounted by up to 95% and advises placing stable instructions and shared reference material first (OpenAI prompt caching).
Both providers set a minimum prompt length below which nothing is cached, and the minimum varies by model.
How to Structure Prompts for Caching
- Static first, dynamic last. Tool definitions, system instructions, policies and reference documents go at the top. The user’s question, the current date and per-request data go at the end.
- Keep the prefix byte-stable. No timestamps, request IDs or randomly ordered lists in the cached part. Serialise JSON with a fixed key order.
- Do not reshuffle tools. Adding, removing or reordering a tool invalidates the cache for everything that follows.
- Measure. API responses report cached and uncached token counts. If cached tokens are zero, the prefix is changing or is too short.
When It Helps and When It Does Not
Caching helps most when many requests share a long, stable beginning: an agent with many tools and a long system prompt, a chat over the same document, a batch of questions against one contract. Agents benefit strongly because every loop step resends the whole conversation, and each step’s prefix is the previous step plus a little more.
It helps little when prompts are short, when each request has a different document, or when requests are spaced further apart than the cache lifetime. Because a cache write can cost more than a normal request, caching content that is used only once can raise the bill.
Prompt Caching vs Other Caches
Prompt caching reuses the processing of the input; the model still generates a new answer. A response cache stores whole answers and returns them for repeated questions, which is cheaper still but only safe when the answer does not depend on changing data or on the user. Semantic caching returns stored answers for similar, not identical, questions and carries a higher risk of returning a wrong one.
In Practice
In agent and assistant projects built by Alltegrio’s AI agent development and LLMOps teams, prompts are assembled in a fixed order from versioned parts, with everything that varies per request at the end. Cache hit rates are tracked per feature on the traces, see AI observability, next to cost and latency. Changes to tool definitions or system prompts are released together, so the cache is invalidated once rather than on every small edit. Prompt structure is part of context engineering, and caching is one of the reasons to treat it as a design decision.