Context engineering → Prompt caching of static prefixes

Prompt Caching Explained: Cut LLM Cost With a Static Prefix

What it is

Every call re-processes the whole prompt. A 4k-token system prompt + 2k of tool definitions + 3k of examples = 9k tokens of identical input on every request. Prompt caching lets the provider store the attention state for that prefix and reuse it: cached input tokens are billed at a steep discount and skip most of the prefill compute, cutting time-to-first-token.

Mechanics (Anthropic as the reference; others are similar in spirit):

  • You mark cache breakpoints (cache_control: {"type": "ephemeral"}) on content blocks; everything up to a breakpoint is a cacheable prefix.
  • Cache hits require the prefix to be byte-identical (same system, same tools, same order, same model).
  • Entries expire after a short TTL (minutes; longer TTL options exist) and are refreshed on use.
  • Minimum cacheable length applies (~1k+ tokens depending on model).
  • Writes cost slightly more than uncached input; reads cost ~10%.

Layering: global stable prefix → per-tenant block → per-conversation history → the new turn. Multiple breakpoints let each layer hit independently.

How to implement

  1. Structure stable-first (prompt structure): system/role, instructions, examples, output schema, tool definitions — all static, all at the front.
  2. Remove variability from the prefix: no timestamps, user names, request IDs, "today is…", random example order, dynamic tool lists. Move them after the breakpoint.
  3. Place breakpoints: after the global system block; after the tenant block (if any); after the conversation history (so multi-turn calls reuse the previous turn's cache); tools are part of the prefix — keep their order fixed.
  4. Same model, same prefix: switching model tiers creates a separate cache; that's fine, just be aware.
  5. Keep sessions warm: for chat, the TTL usually covers the gap between turns; for batch/offline, group work by prompt so the prefix is reused within the TTL window.
  6. Read the usage fields (cache_creation_input_tokens, cache_read_input_tokens) and log them per call (observability & tracing); compute hit rate and savings.
  7. Test cache hits in CI: two consecutive calls with the same prefix must report a cache read; a failing test means someone introduced variability.

Small example

import anthropic, time
client = anthropic.Anthropic()

SYSTEM_STATIC = open("prompts/answer_generate/v3.md").read()          # ~3k tokens, byte-identical
TOOLS = load_tools_sorted()                                            # fixed order

def call(tenant_block: str, history: list[dict], user_turn: str):
    system = [
        {"type": "text", "text": SYSTEM_STATIC, "cache_control": {"type": "ephemeral"}},      # breakpoint 1: global
        {"type": "text", "text": tenant_block,  "cache_control": {"type": "ephemeral"}},      # breakpoint 2: tenant
    ]
    messages = history + [{"role": "user", "content": user_turn}]
    if len(history) >= 2:
        # breakpoint 3: cache the conversation so far (mark the last historical message)
        last = messages[-2]
        last["content"] = [{"type": "text", "text": last["content"] if isinstance(last["content"], str) else last["content"][0]["text"],
                            "cache_control": {"type": "ephemeral"}}]
    t0 = time.time()
    r = client.messages.create(model="claude-sonnet-5", max_tokens=800, system=system, tools=TOOLS, messages=messages)
    u = r.usage
    log_cache(read=u.cache_read_input_tokens, created=u.cache_creation_input_tokens,
              uncached=u.input_tokens, ttft=time.time() - t0)
    return r

# CI probe
def test_cache_hit():
    call("tenant: acme", [], "ping"); r = call("tenant: acme", [], "ping again")
    assert r.usage.cache_read_input_tokens > 0

Pros

  • Cost: 50–90% reduction on input tokens for prompt-heavy products.
  • Latency: large TTFT improvement — the prefill for cached tokens is skipped.
  • Enables richer prompts (more examples, more tools) at little marginal cost.
  • Multi-turn chat and agent loops (same tools/system every step) are ideal workloads.

Cons

  • Fragile to variability: one dynamic token in the prefix silently drops hit rate to zero. You notice on the invoice.
  • TTL: low-traffic prompts expire between calls and pay the write premium.
  • Minimum length: short prompts can't be cached.
  • Provider-specific APIs and semantics; abstractions must expose breakpoints.
  • Cached content still counts toward the context window and still influences attention ("Lost in the middle").

When to use

  • Any call site with ≥ ~1k tokens of static prefix and meaningful volume — system prompts, examples, tool schemas, long tenant instructions, and multi-turn history.
  • Agent loops: every step re-sends the same tools and system prompt.
  • Contextual retrieval ingestion (contextual retrieval): the document is the cached prefix across chunk calls.

When NOT to use

  • Tiny prompts under the minimum.
  • One-off calls with unique prefixes (nothing to reuse).
  • When the "static" content must change per request (per-user instructions that can't be moved after the breakpoint) — restructure first.

Production at scale

Hit rate is a first-class SLI. cache_read / (cache_read + uncached + cache_creation) per call site per prompt version. Alert on drops; a deploy that adds a timestamp to the system prompt is caught in an hour, not a month.

Savings dashboard: tokens read at discount × (full − discounted price) = money saved; show it next to total spend (cost levers).

Layered breakpoints: global (hits across all tenants), tenant (hits within tenant), conversation (hits across turns). Order and count matter: providers limit the number of breakpoints; use them where the reuse is.

Tool definitions are prefix content: keep them sorted, stable, and identical across steps in an agent run (tool design); don't add/remove tools mid-run unless intended (it invalidates the cache from that point).

Warmers for high-value, low-traffic prompts: a scheduled no-op call within the TTL can keep an expensive prefix warm, if the math works (write cost vs read savings). Usually only for very large prefixes.

Batch and offline work: group by prompt/document so consecutive calls share the prefix inside the TTL (contextual retrieval, bulk classification).

Provider abstraction: the LLM client layer must let call sites declare breakpoints; when a fallback provider lacks caching, the cost model changes — reflect it in routing decisions (failure handling).

Security: caches are scoped per organization/API key on major providers; nothing crosses tenants at the provider level. Your own caches (semantic cache, latency levers) must be tenant-scoped.

Evals: caching doesn't change outputs (same tokens → same distribution), but restructuring prompts to be cacheable does; run the eval suite on the restructure.

Prod checklist

  • Stable-first structure; no variability in the prefix (checked in CI)
  • Breakpoints: global / tenant / conversation as applicable; tools sorted and stable
  • Cache usage fields logged per call; hit-rate SLI and alerts per call site/version
  • Savings dashboard
  • Multi-turn: history breakpoint advanced each turn
  • Agent loops: identical tools/system every step
  • Batch work grouped by prefix within TTL
  • Client abstraction exposes breakpoints; fallback cost model adjusted

Metrics to watch

MetricWhy
Cache hit rate per call site/prompt versionThe lever
Cache creation vs read tokensTTL/traffic fit
TTFT with vs without hitsLatency benefit
$ saved by cachingJustification