Chunking → Fixed-size + overlap
What it is
Fixed-size chunking slides a window of a fixed length (measured in tokens, not characters) over the document and emits one chunk per step. The overlap means the last M tokens of chunk i are repeated as the first M tokens of chunk i+1, so a sentence that straddles a boundary is fully present in at least one chunk.
It knows nothing about the document: no headings, no paragraphs, no tables. That is both its strength (works on anything, zero parser dependency) and its weakness (it will cut a table row in half without noticing).
tokens: |----------- 512 -----------|
chunk 1 [############################]
chunk 2 [############################]
|<- 64 ->| overlap
chunk 3 [############################]
How to implement
- Pick a tokenizer that matches your embedding model. A "512-token" chunk measured with the wrong tokenizer may be 700 tokens for the real model and get silently truncated at embed time. Use the embedder's own tokenizer (
tiktokenfor OpenAI models, the HF tokenizer for open models). - Choose
chunk_size. Start at 256–512 tokens for Q&A over prose; 128–256 for very dense reference material; never above the embedder's max sequence length minus a safety margin. - Choose
overlap. 10–20% of chunk size is the usual range. More overlap = more duplicated storage and more near-duplicate hits in top-k. - Tokenize once, slice the token array, decode each slice. Do not re-tokenize per chunk; on a 10M-page corpus that difference is hours.
- Attach metadata to every chunk:
doc_id,chunk_index,char_start,char_end,page,tenant_id(see metadata on every chunk). Without offsets you cannot cite or highlight later. - Make chunk IDs deterministic (
sha256(doc_id + version + chunk_index)), so re-running the pipeline produces the same IDs and upserts instead of duplicating.
Small example
import hashlib
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def fixed_size_chunks(doc_id: str, version: int, text: str,
chunk_size: int = 512, overlap: int = 64):
assert 0 <= overlap < chunk_size
ids = enc.encode(text)
step = chunk_size - overlap
out = []
for i, start in enumerate(range(0, max(len(ids), 1), step)):
window = ids[start:start + chunk_size]
chunk_text = enc.decode(window)
chunk_id = hashlib.sha256(f"{doc_id}:{version}:{i}".encode()).hexdigest()[:16]
out.append({
"chunk_id": chunk_id,
"doc_id": doc_id,
"version": version,
"chunk_index": i,
"token_start": start,
"token_end": start + len(window),
"text": chunk_text,
})
if start + chunk_size >= len(ids):
break
return out
chunks = fixed_size_chunks("policy-123", 3, open("policy.txt").read())
print(len(chunks), chunks[0]["text"][:80])
Pros
- Zero dependency on parsing quality. Works on plain text dumps, logs, transcripts, anything.
- Predictable cost. Number of chunks ≈ tokens / step; storage and embedding spend are easy to forecast.
- Uniform chunk length keeps embedding quality consistent (very short and very long chunks both embed badly).
- Trivially parallel and deterministic; ideal for a first baseline you measure everything else against.
Cons
- Breaks semantics mid-sentence, mid-table, mid-list. The answer to "what is the penalty for late payment" may be split across two chunks, neither of which is retrieved with a high score.
- Overlap duplicates content. Top-k retrieval fills with two near-identical chunks instead of two different facts (see MMR).
- No structural metadata. No breadcrumb, no section title, so a chunk that says "the limit is 5" has no idea what limit.
- Silent truncation risk if tokenizers don't match.
When to use
- As the baseline for every new corpus: build it in an hour, measure recall@k on the golden set, then justify anything fancier by beating this number.
- Unstructured text with no usable structure: chat logs, call transcripts, OCR of receipts, scraped forum posts.
- Very large corpora where cost per chunk dominates and you will rely on reranking (reranking) and parent–child (parent–child) to fix precision.
When NOT to use
- Documents with tables, code, or forms. A table row split in half is worse than no table at all.
- Long structured documents (contracts, manuals, policies) where headings carry the meaning. Use layout / structure-aware layout-aware.
- When you already have a parser that emits structure. Throwing that structure away to do fixed-size is leaving quality on the floor.
Production at scale
Throughput. Chunking is CPU-bound and embarrassingly parallel. Run it as a stage in the ingestion pipeline (queue → worker pool), not inline with the request. A single tokenizer process handles ~1–5 MB/s; batch documents to workers and keep the tokenizer loaded per worker, not per document.
Idempotency and re-ingestion. Deterministic chunk IDs + doc_id + version (see document versioning) mean re-running the pipeline is an upsert, not a duplicate flood. When chunk_size or overlap changes, that is a new chunking config version; store chunking_config_version on each chunk and treat a change like an embedding-model change: dual-index and cut over (versioning & migration).
Cost model. Overlap inflates total embedded tokens by overlap / (chunk_size − overlap). At 512/64 that is +14%; at 256/128 it is +100%. Pick numbers with the invoice in mind.
Retrieval side effects. Because overlapping chunks share text, top-k gets duplicates. Either dedupe by doc_id + adjacent chunk_index after retrieval, or apply MMR. Both are cheap; not doing it wastes context window.
Tenant safety. Chunks are the unit that carries tenant_id and ACLs. If the chunker drops metadata, the index leaks. Assert in tests that every emitted chunk has the mandatory fields.
Observability. Log per-document n_chunks, avg_tokens, and max_tokens. A sudden change in the distribution after a parser upgrade is the earliest signal that something upstream broke.
Failure mode to expect. "Right doc exists, never retrieved" (common failure modes). Fixed-size is the most common cause because the answer sentence was split, or the chunk has the number but not the noun. The fix ladder is: overlap ↑ → recursive splitting → parent–child → contextual retrieval.
Prod checklist
- Tokenizer identical to the embedding model's
-
chunk_size< embedder max length − margin - Deterministic
chunk_id; pipeline is upsert-safe - Every chunk carries
doc_id,version,chunk_index, offsets,tenant_id, ACL -
chunking_config_versionstored; change triggers re-index plan - Post-retrieval dedupe of adjacent overlapping chunks
- Chunk-length distribution dashboard per source
Metrics to watch
| Metric | Why |
|---|---|
| Chunk-level recall@k on golden queries | Detects boundary splits that orphan the answer |
| Tokens embedded / raw tokens | Overlap overhead, cost |
| Duplicate rate in top-k | Overlap side effect |
| p95 chunk tokens | Truncation risk |
Related
- Recursive character splitting — the natural next step
- Parent–child Parent–child — fixes generation starvation without abandoning small chunks
- Chunk metadata — what every chunk must carry
- MMR — removes overlap duplicates from top-k
- Embedding versioning — same discipline applies to chunking config changes