Prompt caching lets LLMs reuse computation for repeated prefixes. New requests sharing a prefix with previously-cached requests skip most of the computation, dramatically reducing cost and latency.
For applications with long static prompts (system prompts, RAG, few-shot examples), prompt caching can cut cost 90%+.
LLMs process prompts as a sequence. Each token's computation depends on all previous tokens (causal attention).
For a prompt of N tokens, the model performs O(N²) work to set up generation.
If two prompts share the first M tokens, the model could reuse the computation for those M tokens — if it has the cached state.
Opt-in via cache_control markers. Cached prefixes:
Mark up to 4 cache breakpoints in your prompt.
Automatic for prompts ≥1024 tokens. No code changes needed.
Explicit context caching. Create cache, reuse it.
vLLM and TGI support prefix caching automatically.
Self-hosting gives the most control.
Long system prompts repeat across requests. Major win.
If your system prompt is 5K tokens and you handle 1M requests/day, that's 5B input tokens/day before user content. Caching collapses this to ~1 cache write + 1M cache reads.
Examples in the prompt repeat. Cache them.
If chunks recur across queries, caching helps.
For typical RAG: each query retrieves different chunks. Caching is more useful for:
Asking many questions about the same document? Cache the document.
Tool schemas in agent prompts can be lengthy. Cache them.
For multi-turn: cache history, append new turn.
If every request has unique content at the start, caching doesn't help.
Below provider thresholds, caching doesn't apply (or doesn't pay back the write cost).
Caching may share infrastructure across requests. Verify provider isolation guarantees.
The key insight: caching works on prefixes. Put stable content first, variable content last.
[user query]
[system prompt]
[examples]
System prompt repeats but isn't a prefix. No caching benefit.
[system prompt]
[examples]
[tool definitions]
[user query]
Stable prefix; variable suffix. Cache hits on every request.
Some providers allow multiple cache breakpoints:
[system prompt] -- cache point 1 (rarely changes)
[examples] -- cache point 2 (occasionally changes)
[tool definitions] -- cache point 3 (changes per app)
[conversation history] -- cache point 4 (grows per conversation)
[user query] -- not cached (always new)
Each cache point can be reused independently when content changes downstream.
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": user_query
}
]
}
]
Cache markers tell Anthropic what to cache. The system measures the prefix up to each marker.
Automatic. Just reuse the same prefix consistently.
Hit rate is typically:
Prefix caching is enabled by default.
Considerations:
For high-traffic deployments, prefix caching dramatically improves throughput.
Track:
If hit rate is low, prompt structure may need work.
Without caching:
With Anthropic caching (90% read discount, 25% write premium):
System prompt has dynamic content (date, user ID) at the start. No cache hits.
Fix: move dynamic content to suffix.
System prompt changes constantly during development. Cache write cost dominates.
Stabilize prompts before relying on caching.
Whitespace differences invalidate cache. Be exactly consistent.
Cache expires at 5 min idle. Bursty traffic patterns may not benefit.
KV cache competes with batch capacity. May need to tune.
Multiple deployment instances may have different caches. Each instance warms separately.
When system prompt changes, all caches invalidate.
For high-volume apps, batch prompt updates rather than rolling out gradually.
Cached requests use less compute but still count against rate limits.
Per-customer caching may be needed. Some providers support isolation; verify.
Caching applies to prefix processing. Streaming the response is independent.
The cost dynamics of LLM inference favor caching. Expect more sophistication.