Chunking → Late chunking

Late Chunking in RAG Explained: Embed First, Cut Later

What it is

Normal ("naive") chunking embeds each chunk independently: the model never sees text outside the chunk, so "it" and "the company" embed as generic tokens. Late chunking (Jina AI, 2024) flips the order:

  1. Feed the entire document (up to the embedder's context limit, e.g. 8k tokens) through the transformer once, producing a contextualised embedding for every token.
  2. Then apply chunk boundaries — to the token embeddings, not the text — and mean-pool the tokens within each chunk into one vector.

Because the transformer's attention already mixed information across the whole document, the token embeddings inside chunk 7 carry knowledge of what chunk 1 defined. The chunk vector for "it must be returned within 14 days" knows "it" is the leased equipment from three paragraphs earlier.

It is the embedding-side answer to the problem that contextual retrieval (contextual retrieval) solves on the text side. No LLM call, no generated text, deterministic.

How to implement

  1. Use an embedding model that supports it: a long-context encoder (8k+) that exposes token-level outputs, e.g. jinaai/jina-embeddings-v3 / v2-base-en, or any HF encoder where you can grab last_hidden_state (nomic-embed, bge-m3 with care).
  2. Tokenize the full document once and record, for each chunk, its (token_start, token_end) span in that tokenization. Chunk boundaries can come from any strategy (recursive, layout-aware); you only need the spans.
  3. Forward pass over the whole document (or the largest window that fits) to get token_embeddings [seq_len, dim].
  4. Pool per span: chunk_vec = mean(token_embeddings[start:end]), then normalise.
  5. Documents longer than the window: process in overlapping macro-windows (e.g. 8k with 512 overlap), and for chunks in the overlap region take the embedding from the window where the chunk is more central.
  6. Index the chunk vectors as usual. Chunk text is unchanged — display and citations are unaffected.
  7. Queries are embedded normally (short, no late chunking needed).

Small example

import torch
from transformers import AutoModel, AutoTokenizer

name = "jinaai/jina-embeddings-v2-base-en"        # 8192-token context, token outputs available
tok = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name, trust_remote_code=True).eval()

def late_chunk_embed(text: str, chunk_char_spans: list[tuple[int, int]]):
    enc = tok(text, return_tensors="pt", return_offsets_mapping=True,
              truncation=True, max_length=8192)
    offsets = enc.pop("offset_mapping")[0]                  # [seq, 2] char offsets per token
    with torch.no_grad():
        token_emb = model(**enc).last_hidden_state[0]       # [seq, dim], context-aware
    vecs = []
    for c_start, c_end in chunk_char_spans:
        mask = (offsets[:, 0] >= c_start) & (offsets[:, 1] <= c_end) & (offsets[:, 1] > 0)
        v = token_emb[mask].mean(dim=0)
        vecs.append(torch.nn.functional.normalize(v, dim=0))
    return torch.stack(vecs)

doc = open("lease.txt").read()
spans = [(0, 1800), (1800, 3600), (3600, 5400)]            # from any chunker; char offsets
chunk_vectors = late_chunk_embed(doc, spans)
print(chunk_vectors.shape)                                 # [3, 768]

Jina's hosted API exposes this as late_chunking=true on the embeddings endpoint.

Pros

  • Document context in every chunk vector, deterministically, with no LLM call and no generated text.
  • Cheaper than contextual retrieval at ingest: one encoder pass per document instead of one LLM call per chunk.
  • Chunker-agnostic: boundaries come from wherever you like; late chunking only changes how vectors are computed.
  • Measurable gains on coreference-heavy text; often close to contextual retrieval's dense-side gain.

Cons

  • Requires a long-context encoder with token-level output; most API embedding endpoints don't expose it. Locks you into a narrower set of models.
  • Whole-document forward passes are expensive in memory: attention over 8k tokens on a base-size model needs a decent GPU; batching long documents is awkward.
  • Documents longer than the window need macro-windowing, which brings back some boundary effects.
  • Doesn't help BM25 at all — the chunk text is unchanged, so keyword search gains nothing (unlike contextual retrieval).
  • Mean-pooling dilutes very long chunks; keep chunks moderately sized.
  • Fewer battle-tested production deployments than the alternatives; treat as a measured experiment.

When to use

  • You self-host embeddings on GPUs and can choose a compatible model.
  • Corpus is coreference-heavy (legal, narrative, technical prose) and you want context-aware chunks without paying for LLM blurbs.
  • Documents mostly fit in the encoder window (≤ 8k tokens) or split into a few macro-windows cleanly.
  • You already use hybrid search and want to lift the dense side specifically.

When NOT to use

  • API-only embedding setups without late-chunking support — you can't do it.
  • Keyword-dominant retrieval (part numbers, IDs, names): late chunking doesn't touch BM25; contextual retrieval does.
  • Tight GPU budgets: long-sequence attention costs grow quickly; naive chunking batches far more efficiently.
  • Corpora of very short documents — there is no cross-chunk context to gain.

Production at scale

GPU planning. Cost is dominated by sequence length, not chunk count. Sort documents by length and batch similar lengths together (bucketed batching) to avoid padding waste. Use FP16/BF16; check memory for the longest window at your batch size. Plan a separate GPU worker pool for embedding; don't co-locate with OCR.

Macro-window strategy. Choose window=8192, stride=7168 (1k overlap). Assign each chunk to the window where it is most central. Log the fraction of chunks that fell in overlap regions; if it's high, your documents are much longer than the window and you should consider hierarchical embeddings or section-level windows from the parser (layout / structure-aware).

Embedding version discipline. Late chunking changes vectors for every chunk. It is an embedding-model migration (versioning & migration): embedding_model_version = "jina-v3+late"; dual-index; compare recall on the golden set; cut over. Changing the chunker also changes spans → recompute pools (cheap if you cache token embeddings; see below).

Cache token embeddings per document version. Store [seq, dim] in FP16 for a limited window (object storage). Then re-chunking (new boundaries) is a pooling operation, not a forward pass — turns a corpus-wide re-embed into a cheap re-pool. At 8k × 768 × 2 bytes ≈ 12 MB per document, this is affordable for hot corpora, not for archives; decide per source.

Hybrid still required. Ship late chunking alongside BM25 (hybrid search) and reranking (reranking). It lifts the dense leg only.

Query-side asymmetry. Queries are embedded normally. Check the model's query/passage instruction prefixes (symmetric vs asymmetric); the mismatch penalty applies here too.

Evaluation before rollout. Run retrieval evals retrieval evals: late vs naive on the same chunks. Adopt per source where it wins; it's not uniformly better.

Vendor lock-in. Only a handful of models support this well. Keep the chunker and index schema independent of the embedding technique so you can revert to naive embeddings with the same model if needed.

Prod checklist

  • Compatible long-context encoder chosen; self-hosted GPU pool sized for max window
  • Bucketed batching by document length
  • Macro-window + stride policy; overlap-region assignment rule
  • Token-embedding cache per doc version (optional but valuable)
  • embedding_model_version reflects late-chunking; dual-index migration plan
  • Hybrid search still in place; reranker still in place
  • Golden-set A/B vs naive embeddings per source

Metrics to watch

MetricWhy
recall@k / nDCG vs naive embeddingsJustifies GPU spend
GPU-seconds per 1k documentsCost
% chunks in macro-window overlapWindow adequacy
Padding waste ratioBatching efficiency