Chunking → Parent–child (small-to-big)

Parent-Child Chunking Explained: Search Small, Answer Big

What it is

There is a built-in tension in chunk size. Small chunks (100–200 tokens) embed precisely and match specific questions well, but they starve the generator: the model sees "the limit is 30 days" without the surrounding clause. Large chunks (1000+ tokens) give the model full context but embed as mush and match poorly.

Parent–child resolves it by indexing two granularities:

  • Child chunks — small, embedded, searched.
  • Parent chunks — larger sections (or the whole document) that contain the children; not embedded (or embedded separately); stored by ID.

At query time you retrieve children, then swap each hit for its parent (deduplicating when multiple children share a parent) and hand the parents to the LLM.

Parent P1 (section, 1200 tok)            Parent P2
 ├─ child c1 (150 tok)  ← embedded        ├─ child c4
 ├─ child c2 (150 tok)  ← embedded        ├─ child c5
 └─ child c3 (150 tok)  ← embedded        └─ child c6

query ──► top-k children: [c2, c5, c3] ──► parents: [P1, P2] ──► LLM

How to implement

  1. Define parents. Best: structural sections from the parser (layout / structure-aware) capped at ~1–2k tokens. Fallback: fixed windows of ~1k tokens. Give each a parent_id.
  2. Define children. Split each parent with recursive (recursive character splitting) or sentence-based splitting into 100–300-token children; each carries parent_id, doc_id, version, tenant_id, ACL.
  3. Embed children only. Store parents in a document store (Postgres, S3, Redis, or the vector store's payload if small) keyed by parent_id.
  4. Retrieve top-k children (k larger than usual, e.g. 20–50, because several will collapse into the same parent).
  5. Collapse to parents. Group hits by parent_id; score a parent by its best child (or sum/max of children); dedupe; take the top-n parents.
  6. Optional rerank (reranking) on children before collapsing, or on parents after — measure both.
  7. Assemble context from parents, within the token budget (per-source truncation budgets). Keep the matched child's text position so you can highlight the exact span in citations (citations).

Variant — sentence-window retrieval: children are single sentences; the "parent" is the sentence ± N neighbours, built at query time from stored offsets. Same idea, finer grain, no separate parent store.

Small example

import hashlib, uuid
from collections import defaultdict

def make_parent_child(doc_id, version, sections, child_size=200):
    """sections: list of {"breadcrumb": str, "text": str} from a structure-aware parser."""
    parents, children = {}, []
    for s_idx, sec in enumerate(sections):
        pid = hashlib.sha256(f"{doc_id}:{version}:{s_idx}".encode()).hexdigest()[:16]
        parents[pid] = {"parent_id": pid, "doc_id": doc_id, "version": version,
                        "breadcrumb": sec["breadcrumb"], "text": sec["text"]}
        for c_idx, ctext in enumerate(recursive_split(sec["text"], chunk_size=child_size)):
            children.append({"chunk_id": f"{pid}:{c_idx}", "parent_id": pid,
                             "doc_id": doc_id, "version": version,
                             "text": f"[{sec['breadcrumb']}] {ctext}"})   # breadcrumb helps embedding
    return parents, children

# --- query time ---
def retrieve_parents(query, vector_store, parent_store, k_children=30, n_parents=5, acl_filter=None):
    hits = vector_store.search(query, k=k_children, filter=acl_filter)   # returns [(chunk_id, parent_id, score)]
    best = defaultdict(float)
    for _, pid, score in hits:
        best[pid] = max(best[pid], score)
    top = sorted(best.items(), key=lambda x: -x[1])[:n_parents]
    return [parent_store.get(pid) | {"score": s} for pid, s in top]

Off-the-shelf: LlamaIndex HierarchicalNodeParser + AutoMergingRetriever; LangChain ParentDocumentRetriever.

Pros

  • Retrieval precision of small chunks + generation context of large chunks. Usually the single biggest quality jump after reranking.
  • Fewer "answer cites the wrong section" errors (common failure modes): the model sees the whole clause, not a fragment.
  • Cheaper context than naive large chunks: you only pay for the parents that actually matched.
  • Works with any child chunker (fixed, recursive, semantic) and any parent definition.
  • Citations improve: you know both the exact matched sentence and its enclosing section.

Cons

  • Two stores to keep consistent. Deleting or re-versioning a document must remove children and parents atomically or you get orphan children pointing to a missing parent (a 404 in the middle of context assembly) or zombie parents from an old version.
  • Bigger context per query. Five parents at 1.2k tokens is 6k tokens of context; budget accordingly (per-source truncation budgets).
  • Parent boundary choice matters. Too big → you're back to context bloat; too small → no benefit.
  • k must be larger at the child level (many children collapse into one parent), which slightly raises vector-search latency.

When to use

  • Almost always for document Q&A once you have a working baseline. This is the upgrade the source file calls highest ROI.
  • Long documents with sections: contracts, manuals, policies, papers, runbooks.
  • When the model gives fragmentary or hedged answers because chunks lack surrounding context.

When NOT to use

  • Corpora of tiny documents (FAQs, product titles, tweets) — document = parent = child; nothing to gain.
  • Strict token budgets on the generation side (tiny context models, cost-critical paths) where large parents don't fit; use sentence-window with a small N instead.
  • When you cannot make the two-store consistency guarantee (e.g. a vector store you can't run transactional deletes against) — solve that first.

Production at scale

Consistency is the whole game. Treat (parents, children) as one write unit keyed by doc_id + version. Ingest order: write parents → write children → flip the "current version" pointer. Delete order: flip pointer → delete children → delete parents. A resumable job (state & durability) with idempotent upserts survives crashes mid-way. Add a nightly orphan check: children whose parent_id is missing, parents with zero children.

Parent storage. Parents are bulky and read by ID, never searched. Put them in Postgres/S3/Redis, not in the vector DB payload — vector stores charge for payload size and slow down with large payloads. Cache hot parents in memory or Redis at query time; a small number of sections answer most questions.

ACL enforcement at both levels. Children carry tenant_id/ACL for the filtered ANN search (authentication & authorization). Parents must carry the same ACL and be re-checked when fetched — a child match under a correct filter followed by a parent fetch with no check is a leak if parent/child ACLs ever drift (e.g. a permission change applied to children only).

Score aggregation. max(child score) is the safe default. sum rewards long parents with many mediocre children. If you rerank (reranking), rerank children (precise), then collapse — reranking 1.2k-token parents is slower and cross-encoders have length limits.

Context assembly. Collapse → dedupe → order by score → fit within the budget (per-source truncation budgets) → put the top parent first or last ("Lost in the middle"). Log which parents made it in and which were cut.

Version transitions. When a doc gets a new version, both old parents and old children must go (soft-delete with version filter, then hard-delete asynchronously). Users asking "what does the current policy say" while old parents linger is the classic incident (document versioning).

Cost. Child embeddings ≈ same token count as any small-chunk strategy. Parents cost storage only. Generation cost rises with parent size — measure $/query before and after; trim n_parents or parent size if it balloons.

Metrics. Chunk-level recall@k (children) and answer faithfulness (parents) should both move. If faithfulness rises but recall doesn't, your children are already fine and the win was context; if recall rises and faithfulness doesn't, parents are too small.

Prod checklist

  • Parent = structural section capped at ~1–2k tokens; child = 100–300 tokens with parent_id
  • Children embedded; parents in a document store keyed by ID
  • Atomic write/delete of parent+child sets per doc_id + version; orphan checker job
  • ACL on both parents and children; parent fetch re-checks
  • k_children ≥ 3× n_parents
  • Score by max child; rerank children before collapse
  • Context budget enforced; matched span preserved for citations
  • Hot-parent cache

Metrics to watch

MetricWhy
Child recall@kRetrieval precision
Answer faithfulness / citation accuracyContext sufficiency
Avg context tokens per queryCost of bigger parents
Orphan children / zombie parents countConsistency
Parent fetch p95 latencyStore choice / caching