Chunking → Fixed-size + overlap

Fixed-Size Chunking in RAG Explained: 512 Tokens, 64 Overlap

What it is

Fixed-size chunking slides a window of a fixed length (measured in tokens, not characters) over the document and emits one chunk per step. The overlap means the last M tokens of chunk i are repeated as the first M tokens of chunk i+1, so a sentence that straddles a boundary is fully present in at least one chunk.

It knows nothing about the document: no headings, no paragraphs, no tables. That is both its strength (works on anything, zero parser dependency) and its weakness (it will cut a table row in half without noticing).

tokens:  |----------- 512 -----------|
chunk 1  [############################]
chunk 2                    [############################]
                           |<- 64 ->|  overlap
chunk 3                                       [############################]

How to implement

  1. Pick a tokenizer that matches your embedding model. A "512-token" chunk measured with the wrong tokenizer may be 700 tokens for the real model and get silently truncated at embed time. Use the embedder's own tokenizer (tiktoken for OpenAI models, the HF tokenizer for open models).
  2. Choose chunk_size. Start at 256–512 tokens for Q&A over prose; 128–256 for very dense reference material; never above the embedder's max sequence length minus a safety margin.
  3. Choose overlap. 10–20% of chunk size is the usual range. More overlap = more duplicated storage and more near-duplicate hits in top-k.
  4. Tokenize once, slice the token array, decode each slice. Do not re-tokenize per chunk; on a 10M-page corpus that difference is hours.
  5. Attach metadata to every chunk: doc_id, chunk_index, char_start, char_end, page, tenant_id (see metadata on every chunk). Without offsets you cannot cite or highlight later.
  6. Make chunk IDs deterministic (sha256(doc_id + version + chunk_index)), so re-running the pipeline produces the same IDs and upserts instead of duplicating.

Small example

import hashlib
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def fixed_size_chunks(doc_id: str, version: int, text: str,
                      chunk_size: int = 512, overlap: int = 64):
    assert 0 <= overlap < chunk_size
    ids = enc.encode(text)
    step = chunk_size - overlap
    out = []
    for i, start in enumerate(range(0, max(len(ids), 1), step)):
        window = ids[start:start + chunk_size]
        chunk_text = enc.decode(window)
        chunk_id = hashlib.sha256(f"{doc_id}:{version}:{i}".encode()).hexdigest()[:16]
        out.append({
            "chunk_id": chunk_id,
            "doc_id": doc_id,
            "version": version,
            "chunk_index": i,
            "token_start": start,
            "token_end": start + len(window),
            "text": chunk_text,
        })
        if start + chunk_size >= len(ids):
            break
    return out

chunks = fixed_size_chunks("policy-123", 3, open("policy.txt").read())
print(len(chunks), chunks[0]["text"][:80])

Pros

  • Zero dependency on parsing quality. Works on plain text dumps, logs, transcripts, anything.
  • Predictable cost. Number of chunks ≈ tokens / step; storage and embedding spend are easy to forecast.
  • Uniform chunk length keeps embedding quality consistent (very short and very long chunks both embed badly).
  • Trivially parallel and deterministic; ideal for a first baseline you measure everything else against.

Cons

  • Breaks semantics mid-sentence, mid-table, mid-list. The answer to "what is the penalty for late payment" may be split across two chunks, neither of which is retrieved with a high score.
  • Overlap duplicates content. Top-k retrieval fills with two near-identical chunks instead of two different facts (see MMR).
  • No structural metadata. No breadcrumb, no section title, so a chunk that says "the limit is 5" has no idea what limit.
  • Silent truncation risk if tokenizers don't match.

When to use

  • As the baseline for every new corpus: build it in an hour, measure recall@k on the golden set, then justify anything fancier by beating this number.
  • Unstructured text with no usable structure: chat logs, call transcripts, OCR of receipts, scraped forum posts.
  • Very large corpora where cost per chunk dominates and you will rely on reranking (reranking) and parent–child (parent–child) to fix precision.

When NOT to use

  • Documents with tables, code, or forms. A table row split in half is worse than no table at all.
  • Long structured documents (contracts, manuals, policies) where headings carry the meaning. Use layout / structure-aware layout-aware.
  • When you already have a parser that emits structure. Throwing that structure away to do fixed-size is leaving quality on the floor.

Production at scale

Throughput. Chunking is CPU-bound and embarrassingly parallel. Run it as a stage in the ingestion pipeline (queue → worker pool), not inline with the request. A single tokenizer process handles ~1–5 MB/s; batch documents to workers and keep the tokenizer loaded per worker, not per document.

Idempotency and re-ingestion. Deterministic chunk IDs + doc_id + version (see document versioning) mean re-running the pipeline is an upsert, not a duplicate flood. When chunk_size or overlap changes, that is a new chunking config version; store chunking_config_version on each chunk and treat a change like an embedding-model change: dual-index and cut over (versioning & migration).

Cost model. Overlap inflates total embedded tokens by overlap / (chunk_size − overlap). At 512/64 that is +14%; at 256/128 it is +100%. Pick numbers with the invoice in mind.

Retrieval side effects. Because overlapping chunks share text, top-k gets duplicates. Either dedupe by doc_id + adjacent chunk_index after retrieval, or apply MMR. Both are cheap; not doing it wastes context window.

Tenant safety. Chunks are the unit that carries tenant_id and ACLs. If the chunker drops metadata, the index leaks. Assert in tests that every emitted chunk has the mandatory fields.

Observability. Log per-document n_chunks, avg_tokens, and max_tokens. A sudden change in the distribution after a parser upgrade is the earliest signal that something upstream broke.

Failure mode to expect. "Right doc exists, never retrieved" (common failure modes). Fixed-size is the most common cause because the answer sentence was split, or the chunk has the number but not the noun. The fix ladder is: overlap ↑ → recursive splitting → parent–child → contextual retrieval.

Prod checklist

  • Tokenizer identical to the embedding model's
  • chunk_size < embedder max length − margin
  • Deterministic chunk_id; pipeline is upsert-safe
  • Every chunk carries doc_id, version, chunk_index, offsets, tenant_id, ACL
  • chunking_config_version stored; change triggers re-index plan
  • Post-retrieval dedupe of adjacent overlapping chunks
  • Chunk-length distribution dashboard per source

Metrics to watch

MetricWhy
Chunk-level recall@k on golden queriesDetects boundary splits that orphan the answer
Tokens embedded / raw tokensOverlap overhead, cost
Duplicate rate in top-kOverlap side effect
p95 chunk tokensTruncation risk