Chunking → Semantic chunking
What it is
Semantic chunking replaces the "every N tokens" rule with a "wherever the meaning changes" rule. The document is split into sentences; each sentence (or a small sliding window of sentences) is embedded; the cosine similarity between consecutive windows is computed; and a chunk boundary is placed where similarity drops below a threshold (absolute, percentile-based, or a local minimum). Sentences between two boundaries are merged into a chunk.
The result is chunks that each talk about one thing, of whatever length that thing takes — which is exactly what an embedding model wants to represent.
sentences: s1 s2 s3 s4 | s5 s6 | s7 s8 s9 s10
similarity: .91 .88 .87 .41 .90 .38 .85 .89 .92
^cut ^cut
How to implement
- Sentence-segment the text with a real segmenter (
pysbd, spaCy,nltk.punkt) — regex on". "is not good enough here because every mistake becomes a candidate boundary. - Window the sentences (e.g. buffer of 1 sentence each side → 3-sentence windows) to smooth noise; single-sentence embeddings are jittery.
- Embed all windows in one batch with a cheap, fast embedder (it does not need to be your retrieval embedder; a small model is fine for boundary detection).
- Compute consecutive cosine distances.
- Pick a boundary rule. Percentile (cut at the top 5–10% largest distances) adapts per document; absolute threshold is stable across documents but needs tuning per corpus; "standard deviation from rolling mean" is a middle ground.
- Enforce size guards.
min_tokens(merge tiny chunks into neighbour) andmax_tokens(force-split long runs with recursive splitting). Without these you get 20-token and 3000-token chunks. - Attach metadata as always; keep sentence offsets so citations work.
- Cache sentence embeddings keyed by
sha256(sentence); re-ingestion of a changed doc then embeds only the new sentences.
Small example
import numpy as np, pysbd
from sentence_transformers import SentenceTransformer
seg = pysbd.Segmenter(language="en", clean=False)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
def semantic_chunks(text, buffer=1, percentile=90, min_tok=80, max_tok=600):
sents = seg.segment(text)
windows = [" ".join(sents[max(0, i-buffer): i+buffer+1]) for i in range(len(sents))]
emb = model.encode(windows, normalize_embeddings=True, batch_size=64)
dist = 1 - np.sum(emb[:-1] * emb[1:], axis=1) # consecutive cosine distance
cut = dist > np.percentile(dist, percentile)
chunks, cur = [], [sents[0]]
for i, is_cut in enumerate(cut, start=1):
cur_len = sum(len(s.split()) for s in cur) # words ≈ tokens, good enough for guards
if (is_cut and cur_len >= min_tok) or cur_len >= max_tok:
chunks.append(" ".join(cur)); cur = []
cur.append(sents[i])
if cur:
chunks.append(" ".join(cur))
return chunks
for i, c in enumerate(semantic_chunks(open("report.txt").read())):
print(i, len(c.split()), c[:70].replace("\n", " "))
Pros
- Topically coherent chunks. Each chunk embeds as one concept, which raises retrieval precision for concept queries.
- Adaptive length. Short topics get short chunks; long explanations stay together.
- No parser needed — works on flat text where structure-aware (layout / structure-aware) cannot.
- Fewer "the answer is split across two chunks" failures than fixed-size on narrative text.
Cons
- Cost. Every sentence gets embedded at ingest, before the real chunk embedding. Ingestion cost roughly doubles versus fixed-size on the embedding line item.
- Non-deterministic-ish. Boundaries depend on the boundary-detection model and its version. Re-running with a new model moves every boundary → re-index.
- Threshold tuning is corpus-specific and brittle: a percentile that works on long reports over-splits FAQs.
- Ignores explicit structure. A heading is a stronger boundary signal than a similarity dip, and semantic chunking doesn't know headings exist.
- Wildly variable sizes without guards; embedding quality degrades at both extremes.
- Evidence of benefit over recursive splitting is mixed on many benchmarks; it often wins on narrative prose and loses on structured/technical text.
When to use
- Narrative, unstructured text with topic drift: long transcripts, meeting notes, research articles converted to plain text, long emails, books.
- When recursive splitting shows measurably poor chunk-level recall and you have no structure to exploit.
- Corpora small enough that doubling ingest embedding cost is irrelevant.
When NOT to use
- Structured documents — use layout-aware (layout / structure-aware); headings beat similarity dips.
- Very large corpora with tight ingestion budgets — the extra embedding pass is real money and real GPU time.
- Tables, code, forms — similarity between rows of a table is meaningless as a boundary signal.
- Short documents / FAQs where each doc is already one topic.
- As a first attempt. Establish fixed-size and recursive baselines first; adopt semantic only if it beats them on your golden set.
Production at scale
Two embedding budgets. Boundary detection and retrieval embedding are separate line items. Use the smallest model that gives stable boundaries for detection (a 33M-param model is fine), keep the good model for retrieval. Track both as separate cost metrics.
Determinism and versioning. Pin the boundary model and its version; store chunking_config_version = {model, percentile, buffer, guards} on every chunk. A change is a corpus-wide re-chunk; plan it like an embedding-model migration (versioning & migration).
Sentence-embedding cache. Keyed by content hash, stored in a KV store. On re-ingestion of a lightly edited document most sentences hit the cache, making incremental updates cheap. This is what makes semantic chunking viable at scale.
Batching and GPU usage. Encode windows in large batches (256–1024) on GPU workers separate from CPU parsing workers. Don't put an embedding model inside every parsing worker; centralize it behind an internal batch endpoint.
Guards are non-negotiable. Enforce min_tokens/max_tokens and log the chunk-length distribution. A percentile rule produces a fixed number of cuts per document regardless of whether the document has topic shifts; FAQ pages get shredded, essays get under-split. Consider a hybrid: structure-aware split first (layout / structure-aware), semantic split only inside oversize sections.
Quality gate. Before adopting, run the chunk-level recall eval (retrieval evals) against recursive splitting on your golden set. Adopt only on sources where it wins; route per source type.
Failure mode to expect. "Costs 10× the estimate" (common failure modes) — someone turned on semantic chunking for a 50M-document corpus and the ingestion GPU bill tripled with no measured recall gain.
Prod checklist
- Real sentence segmenter, locale-aware
- Small pinned boundary model, separate from retrieval embedder
- Percentile/threshold tuned per corpus; stored in versioned config
-
min_tokens/max_tokensguards - Sentence-embedding cache keyed by content hash
- Separate cost metric for boundary embeddings
- A/B against recursive on golden set before rollout; route per source
- Offsets kept for citations
Metrics to watch
| Metric | Why |
|---|---|
| Ingest embedding tokens (boundary vs retrieval) | Cost control |
| Chunk token histogram | Guard effectiveness |
| Chunk-level recall@k vs recursive | Justifies the cost |
| Cache hit rate on re-ingestion | Incremental-update efficiency |
Related
- Recursive — the cheaper baseline it must beat
- Layout-aware — the better choice for structured docs; hybrid option
- Late chunking — a different way to get document context into chunk vectors
- Embedding model choice