Chunking → Metadata on every chunk

Chunk Metadata in RAG Explained: Give Every Chunk a Passport

What it is

Metadata is the structured envelope around the chunk text. The vector store holds (vector, payload); the payload is the metadata. It serves four masters:

ConsumerNeedsExample
Securitytenant, ACL, visibilitytenant_id="acme", allowed_groups=["legal","exec"]
Retrieval qualityfilters, boostsdoc_type="policy", effective_date, language
Citation / UXwhere it came fromsource_url, page, bbox, breadcrumb, title
Operationslineage, migration, debuggingdoc_id, version, chunk_index, parser_version, embedding_model_version, chunking_config_version, ingested_at, content_hash

The rule from the source file is every chunk carries it. Not most chunks. Not "the ones from the main pipeline." Every one.

How to implement

  1. Write the schema first, as a typed model (Pydantic / dataclass / protobuf). Mark fields required vs optional. Required: chunk_id, doc_id, version, tenant_id, acl, chunk_index, text, source, ingested_at, embedding_model_version, chunking_config_version.
  2. Populate at each stage, never later. Connector sets source, source_url, tenant_id, acl, modified_at, content_hash. Parser sets page, bbox, element_type, parser_version. Chunker sets chunk_index, breadcrumb, section, offsets, chunking_config_version. Embedder sets embedding_model_version.
  3. Validate before write. Reject the chunk (and fail the document loudly) if a required field is missing. A chunk with no tenant_id must never reach the index.
  4. Split payload by access pattern. Small, filterable fields go in the vector store payload (tenant, ACL, doc_type, dates, doc_id). Bulky, display-only fields (full breadcrumb text, table HTML, bboxes) go to a document store keyed by chunk_id.
  5. Index the filter fields. Most vector stores need explicit payload indexes for fast filtering (Qdrant create_payload_index, pgvector B-tree/GIN on the column, Pinecone metadata config).
  6. Normalise types. Dates as ISO-8601 strings and as epoch ints for range filters; enums as lowercase strings; lists for multi-valued ACLs.
  7. Emit metadata into the citation at generation time: [title, section, page, url].

Small example

from pydantic import BaseModel, Field, field_validator
from datetime import datetime

class ChunkMeta(BaseModel):
    # identity & lineage
    chunk_id: str
    doc_id: str
    version: int
    chunk_index: int
    content_hash: str
    ingested_at: datetime
    parser_version: str
    chunking_config_version: str
    embedding_model_version: str
    # security  (required — no default!)
    tenant_id: str
    acl_groups: list[str] = Field(min_length=1)
    visibility: str            # "internal" | "public" | "restricted"
    # retrieval filters
    source: str                # "confluence" | "s3" | "sharepoint"
    doc_type: str
    language: str = "en"
    effective_date: datetime | None = None
    # citation / display
    title: str
    breadcrumb: str = ""
    page: int | None = None
    source_url: str | None = None
    char_start: int
    char_end: int

    @field_validator("tenant_id")
    @classmethod
    def non_empty(cls, v):
        if not v.strip():
            raise ValueError("tenant_id required")
        return v

def to_payload(m: ChunkMeta) -> dict:
    """Lean payload for the vector store; bulky fields live elsewhere keyed by chunk_id."""
    return {
        "doc_id": m.doc_id, "version": m.version, "tenant_id": m.tenant_id,
        "acl_groups": m.acl_groups, "visibility": m.visibility, "doc_type": m.doc_type,
        "source": m.source, "language": m.language,
        "effective_ts": int(m.effective_date.timestamp()) if m.effective_date else None,
        "page": m.page, "title": m.title, "emb_v": m.embedding_model_version,
    }

# vector_store.upsert(id=m.chunk_id, vector=vec, payload=to_payload(m))
# doc_store.put(m.chunk_id, {"text": text, "breadcrumb": m.breadcrumb, "bbox": bbox, ...})

Query-time filter (Qdrant-style):

flt = {"must": [
    {"key": "tenant_id", "match": {"value": caller.tenant}},
    {"key": "acl_groups", "match": {"any": caller.groups}},
    {"key": "version_current", "match": {"value": True}},
]}
hits = store.search(query_vec, limit=30, query_filter=flt)

Pros

  • Security becomes enforceable at the index, not in a prompt (authentication & authorization).
  • Filters ("policies effective after 2024", "only Confluence") turn a hopeless semantic search into a precise one (self-query).
  • Citations with page and URL make answers verifiable (citations).
  • Migrations (embedding model, parser, chunker) become tractable because every chunk states which versions produced it (versioning & migration).
  • Debugging a bad answer is possible: you can trace chunk → doc → version → source → parser.

Cons

  • Schema discipline is work. Every connector and parser must map into it; every new source needs an adapter.
  • Payload bloat slows vector stores and raises cost if you dump everything into the payload.
  • Schema evolution — adding a required field to a 100M-chunk index means a backfill job.
  • Filter cardinality issues: a filter that matches 0.01% of chunks can make ANN degrade (filtered search).

When to use

Always. There is no scenario where chunks should be indexed without identity, version, tenant, and ACL. The only question is how rich the optional metadata gets.

When NOT to use

Not applicable — but do not over-index: don't put free-text fields, full breadcrumb strings, or bboxes in the filterable payload. Keep those in the document store.

Production at scale

ACL is a required field with no default. The single most important line in the schema. The number of production leaks caused by "the new connector didn't set tenant_id so it defaulted to null and null matched everything" is not small. Make the validator reject, make the pipeline fail loudly, and add a nightly scan for chunks with missing/empty security fields.

Group-based ACLs, not user-based. Store acl_groups (or a document-level ACL ID that resolves to groups), not user IDs. Permissions change constantly; re-indexing chunks on every user change is unworkable. At query time resolve the caller → groups and filter acl_groups ∩ caller_groups ≠ ∅. For very dynamic permissions, store acl_id only and keep the resolution in a fast side store; still filter at query time, never post-hoc (authentication & authorization).

Version pointer. Mark the current version with is_current=true or filter on version = current_version(doc_id) via a join (pgvector) or a payload field updated on flip (vector DBs). Soft-delete old versions (document versioning), hard-delete asynchronously.

Payload indexes. Every field you filter on needs an index or you get a scan. Check the store's docs; measure filtered-search p95 with realistic selectivity.

Lineage fields drive migrations. embedding_model_version lets you run a dual-index cutover by filtering on it. parser_version lets you find and re-parse only documents from a bad parser release. content_hash lets change detection skip unchanged content (change detection).

Schema versioning. Add meta_schema_version. Additive changes (new optional field) need no backfill; new required fields need a backfill job with a progress checkpoint (state & durability). Never change the meaning of an existing field.

Denormalise doc-level fields onto chunks. title, doc_type, effective_date are per document, but the vector store filters per chunk. Copy them down. Keep the document table as source of truth; re-sync on document update.

Multi-region / residency. region as a field, plus physical separation of indexes per region where required (data residency & retention). Metadata cannot substitute for physical separation when regulation demands it.

Observability. Dashboards: chunks missing optional fields by source; distinct embedding_model_version values live (should be 1 after a migration); chunks with is_current=false older than the hard-delete SLA.

Prod checklist

  • Typed schema with required security + lineage fields; validation before write
  • Group-based ACLs; query-time filter with caller identity
  • Lean filterable payload; bulky fields in document store by chunk_id
  • Payload indexes on every filter field; filtered p95 measured
  • Version pointer + soft-delete; hard-delete job
  • meta_schema_version; backfill procedure for new required fields
  • Nightly scan for missing/empty security fields
  • Citation renderer consumes title/section/page/url

Metrics to watch

MetricWhy
Chunks with missing required fields (should be 0)Leak prevention
Filtered search p95 vs unfilteredIndex adequacy
Live embedding_model_version countMigration hygiene
Stale (non-current) chunks past SLAVersioning hygiene