Chunking → Recursive character splitting
What it is
Recursive splitting takes an ordered list of separators — typically ["\n\n", "\n", ". ", " ", ""] — and applies them hierarchically. It first splits the text on the coarsest separator (paragraphs). Any resulting piece larger than chunk_size is split again with the next separator (lines), then sentences, then words, and finally characters as a last resort. Small adjacent pieces are merged back together until they approach chunk_size.
The effect is that chunk boundaries fall on paragraph breaks whenever possible, on sentence breaks when a paragraph is too long, and only rarely mid-sentence. You get most of the semantic benefit of "smart" chunking with none of the parser dependency.
How to implement
- Define separators per content type. Prose:
["\n\n", "\n", ". ", " "]. Markdown: add"\n## ","\n### "at the front. Code: split on function/class boundaries first ("\nclass ", "\ndef "). Store this as config, versioned. - Measure length in tokens using the embedder's tokenizer (same rule as fixed-size + overlap).
- Split → recurse → merge. Split on separator[0]; for each piece over budget, recurse with separator[1:]; then greedily merge consecutive pieces while the merged size ≤
chunk_size. - Keep the separator in the text (or at least the whitespace) so decoded chunks read naturally and sentence boundaries survive.
- Optionally add overlap at the sentence level (repeat the last sentence of the previous chunk) instead of raw token overlap. It duplicates less and preserves more.
- Emit the same metadata as any chunker:
doc_id,version,chunk_index, offsets,tenant_id, ACL.
Small example
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
tok_len = lambda s: len(enc.encode(s))
SEPARATORS = ["\n\n", "\n", ". ", " ", ""]
def recursive_split(text: str, chunk_size: int = 400, seps=SEPARATORS) -> list[str]:
if tok_len(text) <= chunk_size or not seps:
return [text]
sep, rest = seps[0], seps[1:]
pieces = text.split(sep) if sep else list(text)
# re-attach separator so text stays readable
pieces = [p + sep for p in pieces[:-1]] + [pieces[-1]]
chunks, buf = [], ""
for p in pieces:
if tok_len(p) > chunk_size: # piece itself too big → recurse
if buf:
chunks.append(buf); buf = ""
chunks.extend(recursive_split(p, chunk_size, rest))
elif tok_len(buf + p) <= chunk_size: # fits → merge
buf += p
else: # doesn't fit → flush
chunks.append(buf); buf = p
if buf:
chunks.append(buf)
return [c for c in chunks if c.strip()]
doc = open("handbook.md").read()
for i, c in enumerate(recursive_split(doc)):
print(i, tok_len(c), repr(c[:60]))
Equivalent off-the-shelf: langchain_text_splitters.RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=400, chunk_overlap=40).
Pros
- Respects natural boundaries the majority of the time, so far fewer split sentences than fixed-size.
- Still parser-independent. Works on plain text, Markdown, HTML-to-text output.
- Configurable per content type by changing the separator list only.
- Cheap. Pure string ops; same throughput class as fixed-size.
- Good default for prose corpora: wikis, tickets, emails, docs.
Cons
- Variable chunk sizes. A tiny paragraph becomes a tiny chunk with a weak embedding; a wall-of-text paragraph becomes a chunk cut at sentence boundaries that may still separate a claim from its qualifier.
- Separators are heuristics.
". "splits "Dr. Smith" and "v1.2. Released". Bullet lists with no blank lines merge into one blob. - Still blind to tables and headings unless you add them as separators — at which point you are halfway to layout / structure-aware.
- Merging logic can orphan context: a heading line may end up as the last line of the previous chunk rather than the first line of the section it titles.
When to use
- Default choice for prose-heavy corpora where the parser gives you text but not rich structure.
- Markdown / lightly-structured content where heading separators are easy to add.
- When you want better boundaries than fixed-size without introducing an embedding-based (semantic chunking) or parser-based (layout / structure-aware) dependency.
When NOT to use
- Tables, forms, spreadsheets, code with long functions — boundaries do not fall where the semantics do.
- Documents whose parser already emits structural elements (Docling, Unstructured). Use those elements directly (layout / structure-aware).
- Very short documents (tweets, ticket titles): one document = one chunk; no splitting needed.
Production at scale
Separator config is a versioned artifact. Changing ". " to a regex sentence splitter changes every chunk boundary in the corpus → every chunk ID → a full re-embed. Store chunking_config_version on chunks, keep the config in git, and gate changes behind the eval suite exactly like a prompt change (prompt versioning & evaluation).
Heading-orphan bug (real, common). Because merging is greedy, a section heading often gets glued to the tail of the previous chunk. Fix: treat heading separators as keep-with-next — split before them and never merge across them. Or move to layout-aware chunking where headings become metadata (layout / structure-aware, headings hierarchy).
Size floor and ceiling. Enforce min_tokens (merge tiny fragments into neighbours) and max_tokens (hard cap below embedder limit). Log the distribution; a bimodal histogram usually means a source has a different newline convention (\r\n, single-newline paragraphs) and needs its own separator list.
Language and locale. ". " fails for Chinese/Japanese (no space after 。), and for languages with different punctuation. Route by detected language to a locale-specific separator set, or use a sentence-segmentation model (pysbd, spaCy) for the sentence level.
Throughput. The recursion is cheap; the token counting is not. Cache tok_len on the piece string or count once on the token array and slice. On large corpora, run chunking in the same worker that parsed the document to avoid shipping text twice.
Overlap strategy at scale. Sentence-level overlap (repeat last sentence) is 1–3% extra tokens; raw token overlap of 20% is 20% extra spend. At tens of millions of chunks the difference is real money.
Debuggability. Store char_start/char_end offsets into the parsed text. When a golden query fails, you must be able to open the doc and see exactly where the boundary fell.
Prod checklist
- Separator list per content type, in config, versioned
- Token-based length with the embedder's tokenizer
- Keep-with-next rule for headings (or move to layout-aware)
-
min_tokens/max_tokensenforced; distribution monitored - Locale-aware sentence splitting for non-English sources
- Offsets stored for every chunk
- Config change → re-index plan with dual-index cutover
Metrics to watch
| Metric | Why |
|---|---|
| % chunks ending mid-sentence | Separator quality |
| Chunk token histogram (p5, p50, p95) | Fragment and oversize detection |
| Heading-orphan rate (heading is last line of a chunk) | Merge bug |
| recall@k vs fixed-size baseline | Is it actually better on your corpus? |
Related
- Fixed-size — the baseline to beat
- Layout-aware — when the parser gives you structure
- Parent–child Parent–child — pair with small recursive chunks for best precision
- Headings hierarchy — where heading breadcrumbs come from