Chunking → Contextual retrieval (LLM-generated chunk context)

Contextual Retrieval in RAG Explained: Fix Chunks That Lose Context

What it is

A chunk in isolation often loses the information that makes it findable. "Revenue grew 3% over the previous quarter" — which company, which quarter? "It must be returned within 14 days" — what must? The original document made that clear from context; the chunk didn't keep it.

Contextual retrieval (popularised by Anthropic, 2024) fixes this at ingest: for each chunk, an LLM is given the whole document (or a large window of it) plus the chunk, and asked to produce a short context blurb. That blurb is prepended to the chunk text before it is embedded and before it is BM25-indexed. Reported reductions in retrieval failure rate: roughly 35% with embeddings alone, ~49% when combined with contextual BM25, ~67% when reranking is added on top.

It complements breadcrumbs (layout / structure-aware): a breadcrumb gives the heading path; the LLM blurb gives what the chunk is about in the document's own terms, including entities that the heading omits.

How to implement

  1. Pick the context source. Whole document if it fits the model's window; otherwise the enclosing section + document title + abstract/first page. Use prompt caching (prompt caching of static prefixes): the document is the cached prefix, each chunk is the small varying suffix — this is what makes it affordable.
  2. Prompt (temperature 0):

    <document>…</document> Here is the chunk we want to situate: <chunk>…</chunk>. Give a short, succinct context (1–2 sentences) to situate this chunk within the overall document for the purposes of improving search retrieval. Answer only with the context.

  3. Prepend context + "\n\n" + chunk_text to form the indexed text. Store the original chunk text separately for display (users shouldn't see the blurb, or should see it marked).
  4. Embed the enriched text; BM25-index the enriched text too (hybrid search).
  5. Use a cheap, fast model (Haiku-class). Quality of blurbs matters less than presence of the right entities.
  6. Cache by sha256(doc_version + chunk_text) so re-ingestion doesn't regenerate.
  7. Validate: blurb length cap (≤ 100 tokens), non-empty, no refusal text, language matches document.

Small example

import anthropic, hashlib
client = anthropic.Anthropic()

PROMPT = """<document>
{doc}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Please give a short succinct context (1-2 sentences) to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk. Answer only with the context."""

def contextualize(doc_text: str, chunks: list[str], cache: dict) -> list[str]:
    enriched = []
    for ch in chunks:
        key = hashlib.sha256((doc_text[:64] + ch).encode()).hexdigest()
        if key not in cache:
            resp = client.messages.create(
                model="claude-haiku-4-5-20251001",
                max_tokens=120,
                temperature=0,
                system=[{"type": "text", "text": "You write terse retrieval context.",
                         "cache_control": {"type": "ephemeral"}}],
                messages=[{"role": "user", "content": [
                    {"type": "text", "text": f"<document>\n{doc_text}\n</document>",
                     "cache_control": {"type": "ephemeral"}},          # document prefix cached across chunks
                    {"type": "text", "text": PROMPT.split("</document>")[1].format(chunk=ch)},
                ]}],
            )
            cache[key] = resp.content[0].text.strip()
        enriched.append(cache[key] + "\n\n" + ch)
    return enriched

Pros

  • Large, measured recall gains on chunks full of pronouns, bare numbers, relative references, and entity-less sentences.
  • Helps BM25 as much as dense: the blurb adds the entity names and keywords the chunk lacked.
  • Chunker-agnostic: works on top of fixed, recursive, layout-aware, or parent–child children.
  • Cheap at scale with prompt caching — the document is cached; each chunk costs only the small suffix + ~50 output tokens.

Cons

  • Ingest cost and latency rise. One LLM call per chunk. Without caching, cost is O(doc_tokens × n_chunks) — prohibitive. With caching it's roughly O(doc_tokens + n_chunks × 100).
  • Non-deterministic across model versions; re-generation changes indexed text → re-embed.
  • Blurb errors propagate. A hallucinated entity in the context makes the chunk retrievable for the wrong query. Rare, but the failure is silent.
  • Doesn't help if parsing is broken — the LLM can only situate a chunk in a document it can read.
  • Adds a third version dimension (context-model version) alongside chunking config and embedding model.

When to use

  • Corpora where chunks are context-poor: financial reports, legal clauses, technical specs with lots of "it", "this section", "the above", tables of numbers.
  • When the failure bucket "right doc exists, never retrieved" (common failure modes) is large and hybrid search alone didn't fix it.
  • Long documents whose heading breadcrumbs are uninformative ("Section 4.2.1").

When NOT to use

  • Chunks that are already self-contained (FAQ entries, product descriptions, short tickets): no gain, pure cost.
  • Hard ingest budgets or real-time ingestion SLAs where an LLM call per chunk is unaffordable or too slow — or restrict it to high-value sources only.
  • Highly regulated corpora where you cannot send document text to a third-party model — use a self-hosted small model or skip.
  • Before hybrid search and reranking exist. Those are cheaper wins; do them first.

Production at scale

Cost math first. Per document: doc_tokens (cached after first chunk) + n_chunks × (chunk_tokens + ~50 out). For a 10k-token doc with 50 chunks on a Haiku-class model with 90% cache discount, this is cents, not dollars. Multiply by corpus size and by re-ingestion rate — the second number is what surprises people. Put a per-source budget and a kill switch on it.

Batch, don't stream. Run contextualization as an async batch stage with the provider's batch API (50% cheaper on most providers) for backfills; use the online path only for incremental updates that must be searchable quickly.

Cache aggressively. Key: sha256(model_version + doc_version + chunk_text). Re-ingesting a doc where three paragraphs changed regenerates three blurbs, not fifty. Combine with change detection (change detection) so unchanged docs never reach this stage.

Selective application. Not every source needs it. Route by source type or by a cheap heuristic (chunk has < N named entities, or > M pronouns per sentence). Log which chunks were enriched.

Versioning. Store context_model_version on the chunk. A model swap changes indexed text for the whole corpus. Treat it like an embedding-model migration (versioning & migration): dual-index, evaluate, cut over. Don't let a provider deprecation force an unplanned corpus-wide regeneration — pin the model.

Data governance. The document goes to an LLM. Confirm zero-retention endpoints, region, and tenant consent (data residency & retention). For tenant data, run per-tenant batches so one tenant's document never sits in a cache alongside another's (prompt caches are per-organization on major providers, but internal caches you build must be tenant-scoped).

Display vs index text. Keep text_display (original) and text_indexed (context + original) as separate fields. Show users the original; cite by original offsets. The blurb is an index artifact, not content.

Quality gate. Sample 1% of blurbs into a review queue. Check: length, language, no hallucinated entities (blurb entities ⊆ document entities is a cheap automatic check), no refusals. Track failure rate per source; a spike after a model update is your alarm.

Combine with hybrid + rerank. The published gains compound: contextual embeddings + contextual BM25 + rerank. Don't ship contextual retrieval into a dense-only pipeline and expect the headline number.

Prod checklist

  • Prompt caching on the document prefix; batch API for backfills
  • Cache keyed by model + doc version + chunk hash
  • Per-source enablement and budget; kill switch
  • context_model_version on chunks; migration plan for model changes
  • text_display vs text_indexed separated; citations use original offsets
  • Blurb validation (length, language, entity-subset check, no refusal)
  • Zero-retention / region compliance confirmed for the context model
  • Enriched text fed to both dense and BM25 indexes

Metrics to watch

MetricWhy
Retrieval failure rate (recall@20 misses) before/afterThe headline gain
$ per document contextualized; cache hit rateCost control
Blurb validation failure rateSilent quality drift
Ingest p95 latency per docSLA impact