Knowing that prompt caching exists is the easy part. Designing prompts to maximize cache hits across realistic workloads takes practice.
This page goes deeper than the mechanics into the strategy.
Caching is prefix-based. Maximize cache hits by:
This single principle drives most of the strategy.
Think in layers from most stable to most variable:
| Layer | Stability | Update frequency |
|---|---|---|
| Constants | very stable | rarely |
| System prompt | stable | days/weeks |
| Examples | semi-stable | weeks/months |
| Tool definitions | semi-stable | per-app |
| Document context | per-conversation | per-conversation |
| Conversation history | grows | per-turn |
| Current query | unique | per-request |
Each layer can have its own cache breakpoint (where supported).
Some content seems variable but can be stabilized:
Don't put exact timestamp in prompt. Use coarse "2026-04" or omit if not essential.
If only used for personalization, move to dedicated section after the cacheable prefix.
Examples chosen at random invalidate the cache. Use stable example sets.
Push to suffix where possible.
Always organize as:
[stable prefix - long, cacheable]
---
[volatile suffix - short, unique]
This works regardless of provider.
For applications with multiple stable layers, use multiple cache breakpoints.
Anthropic supports up to 4 cache control markers.
[system: 5K tokens] --- cache 1
[examples: 3K tokens] --- cache 2
[tools: 2K tokens] --- cache 3
[user input: variable] (not cached)
When examples change, cache 1 still hits.
When system changes, all caches invalidate.
Order from most-stable to least.
For chatting about a document:
[system prompt]
[document content] --- cache here
[turn 1]
[turn 2]
...
The document acts as a long stable suffix. Each new turn benefits from caching the document.
Few-shot examples are great for quality but bloat prompts.
Use a fixed set of N examples. Cache them.
Selecting examples per query (e.g., from a few-shot index) defeats caching.
Compromise: cluster queries; cache examples per cluster.
Avoid reformatting examples without need. Whitespace changes invalidate.
Keep system prompts in version control. Treat changes as deployments.
Bad: editing the system prompt for each conversation.
Good: stable system prompt; per-conversation parameters in conversation.
Use cases where instructions vary by user/context:
Bad placement (defeats caching):
You are helping {user_name}, who prefers {communication_style}...
[long stable content]
[query]
Better:
[long stable content]
User context: name={user_name}, style={communication_style}
Query: ...
The user-specific bits move to the variable region.
For multi-turn conversations:
[system + tools]
[turn 1 user]
[turn 1 assistant]
[turn 2 user]
[turn 2 assistant]
[turn 3 user] <-- new
Each turn extends the cached prefix. Most providers cache up to the last assistant message.
For long conversations, cache hits stay high until conversation grows beyond cache TTL.
Caches expire (5 min default). For low-traffic apps, the cache may be cold by next request.
Mitigation:
Reordering tool definitions defeats caching. Sort consistently.
Different trailing whitespace = different cache key.
Generating prompts via templates that vary subtly by environment.
Adding "Today is X" each request invalidates the prefix.
Some apps shuffle examples by user_id hash. Defeats caching.
Key metrics:
Hit rate by request type. Investigate low-hit categories.
Average count tells you how much work is being saved.
Compare cached cost to uncached cost. Some apps see 90%+ reduction.
Cache hits also reduce time to first token.
Per-customer system prompt with policies + tools + examples.
Cache: everything except current ticket text. ~95% hits.
System + recent code context.
Cache: system + opened files. New question = new suffix.
Document set + history + new query.
Cache: document set (if same per session) + history.
System + tools + ReAct trace so far.
Cache: extends with each action.
Cache pays back when:
(uncached_cost - cached_cost) × hits > write_cost × cache_lifecycle
For long prompts and high reuse: ratio is dramatic.
For short prompts: write premium may exceed savings.
When cache hit rate is lower than expected:
Hit rate should approach 95%+ for well-structured prompts.
Tool results in conversation history affect the prefix from that point on.
Stable formatting helps caching subsequent turns.
Caching is on prefix processing. Streaming output independent.
Failed turns may leave inconsistent state. Decide whether to retry from cached prefix or restart.
Cache is typically per-region. Failover may invalidate.
For now: structure prompts deliberately. Big payoff for moderate effort.