Context engineering → Prompt caching of static prefixes
What it is
Every call re-processes the whole prompt. A 4k-token system prompt + 2k of tool definitions + 3k of examples = 9k tokens of identical input on every request. Prompt caching lets the provider store the attention state for that prefix and reuse it: cached input tokens are billed at a steep discount and skip most of the prefill compute, cutting time-to-first-token.
Mechanics (Anthropic as the reference; others are similar in spirit):
- You mark cache breakpoints (
cache_control: {"type": "ephemeral"}) on content blocks; everything up to a breakpoint is a cacheable prefix. - Cache hits require the prefix to be byte-identical (same system, same tools, same order, same model).
- Entries expire after a short TTL (minutes; longer TTL options exist) and are refreshed on use.
- Minimum cacheable length applies (~1k+ tokens depending on model).
- Writes cost slightly more than uncached input; reads cost ~10%.
Layering: global stable prefix → per-tenant block → per-conversation history → the new turn. Multiple breakpoints let each layer hit independently.
How to implement
- Structure stable-first (prompt structure): system/role, instructions, examples, output schema, tool definitions — all static, all at the front.
- Remove variability from the prefix: no timestamps, user names, request IDs, "today is…", random example order, dynamic tool lists. Move them after the breakpoint.
- Place breakpoints: after the global system block; after the tenant block (if any); after the conversation history (so multi-turn calls reuse the previous turn's cache); tools are part of the prefix — keep their order fixed.
- Same model, same prefix: switching model tiers creates a separate cache; that's fine, just be aware.
- Keep sessions warm: for chat, the TTL usually covers the gap between turns; for batch/offline, group work by prompt so the prefix is reused within the TTL window.
- Read the usage fields (
cache_creation_input_tokens,cache_read_input_tokens) and log them per call (observability & tracing); compute hit rate and savings. - Test cache hits in CI: two consecutive calls with the same prefix must report a cache read; a failing test means someone introduced variability.
Small example
import anthropic, time
client = anthropic.Anthropic()
SYSTEM_STATIC = open("prompts/answer_generate/v3.md").read() # ~3k tokens, byte-identical
TOOLS = load_tools_sorted() # fixed order
def call(tenant_block: str, history: list[dict], user_turn: str):
system = [
{"type": "text", "text": SYSTEM_STATIC, "cache_control": {"type": "ephemeral"}}, # breakpoint 1: global
{"type": "text", "text": tenant_block, "cache_control": {"type": "ephemeral"}}, # breakpoint 2: tenant
]
messages = history + [{"role": "user", "content": user_turn}]
if len(history) >= 2:
# breakpoint 3: cache the conversation so far (mark the last historical message)
last = messages[-2]
last["content"] = [{"type": "text", "text": last["content"] if isinstance(last["content"], str) else last["content"][0]["text"],
"cache_control": {"type": "ephemeral"}}]
t0 = time.time()
r = client.messages.create(model="claude-sonnet-5", max_tokens=800, system=system, tools=TOOLS, messages=messages)
u = r.usage
log_cache(read=u.cache_read_input_tokens, created=u.cache_creation_input_tokens,
uncached=u.input_tokens, ttft=time.time() - t0)
return r
# CI probe
def test_cache_hit():
call("tenant: acme", [], "ping"); r = call("tenant: acme", [], "ping again")
assert r.usage.cache_read_input_tokens > 0
Pros
- Cost: 50–90% reduction on input tokens for prompt-heavy products.
- Latency: large TTFT improvement — the prefill for cached tokens is skipped.
- Enables richer prompts (more examples, more tools) at little marginal cost.
- Multi-turn chat and agent loops (same tools/system every step) are ideal workloads.
Cons
- Fragile to variability: one dynamic token in the prefix silently drops hit rate to zero. You notice on the invoice.
- TTL: low-traffic prompts expire between calls and pay the write premium.
- Minimum length: short prompts can't be cached.
- Provider-specific APIs and semantics; abstractions must expose breakpoints.
- Cached content still counts toward the context window and still influences attention ("Lost in the middle").
When to use
- Any call site with ≥ ~1k tokens of static prefix and meaningful volume — system prompts, examples, tool schemas, long tenant instructions, and multi-turn history.
- Agent loops: every step re-sends the same tools and system prompt.
- Contextual retrieval ingestion (contextual retrieval): the document is the cached prefix across chunk calls.
When NOT to use
- Tiny prompts under the minimum.
- One-off calls with unique prefixes (nothing to reuse).
- When the "static" content must change per request (per-user instructions that can't be moved after the breakpoint) — restructure first.
Production at scale
Hit rate is a first-class SLI. cache_read / (cache_read + uncached + cache_creation) per call site per prompt version. Alert on drops; a deploy that adds a timestamp to the system prompt is caught in an hour, not a month.
Savings dashboard: tokens read at discount × (full − discounted price) = money saved; show it next to total spend (cost levers).
Layered breakpoints: global (hits across all tenants), tenant (hits within tenant), conversation (hits across turns). Order and count matter: providers limit the number of breakpoints; use them where the reuse is.
Tool definitions are prefix content: keep them sorted, stable, and identical across steps in an agent run (tool design); don't add/remove tools mid-run unless intended (it invalidates the cache from that point).
Warmers for high-value, low-traffic prompts: a scheduled no-op call within the TTL can keep an expensive prefix warm, if the math works (write cost vs read savings). Usually only for very large prefixes.
Batch and offline work: group by prompt/document so consecutive calls share the prefix inside the TTL (contextual retrieval, bulk classification).
Provider abstraction: the LLM client layer must let call sites declare breakpoints; when a fallback provider lacks caching, the cost model changes — reflect it in routing decisions (failure handling).
Security: caches are scoped per organization/API key on major providers; nothing crosses tenants at the provider level. Your own caches (semantic cache, latency levers) must be tenant-scoped.
Evals: caching doesn't change outputs (same tokens → same distribution), but restructuring prompts to be cacheable does; run the eval suite on the restructure.
Prod checklist
- Stable-first structure; no variability in the prefix (checked in CI)
- Breakpoints: global / tenant / conversation as applicable; tools sorted and stable
- Cache usage fields logged per call; hit-rate SLI and alerts per call site/version
- Savings dashboard
- Multi-turn: history breakpoint advanced each turn
- Agent loops: identical tools/system every step
- Batch work grouped by prefix within TTL
- Client abstraction exposes breakpoints; fallback cost model adjusted
Metrics to watch
| Metric | Why |
|---|---|
| Cache hit rate per call site/prompt version | The lever |
| Cache creation vs read tokens | TTL/traffic fit |
| TTFT with vs without hits | Latency benefit |
| $ saved by caching | Justification |
Related
- Prompt structure — stable-first
- Budgets — cached tokens still count toward the window
- Contextual retrieval — caching at ingest
- Cost levers, Latency levers
- Agent loop — same prefix each step