Source ingestion → Document versioning
What it is
A document is not a static thing. Policies get amended, contracts get renewed, wiki pages get edited, prices change. Each edit produces a new version. If the index just contains "chunks of the doc", then after an edit it contains chunks of both versions until someone cleans up — and the old chunks, embedded and ranked identically, answer questions with outdated facts.
Versioning makes every chunk say which version it belongs to, makes queries filter to the current version, and makes cleanup safe and asynchronous:
doc_id = "policy-42"
v3 (current=true) chunks c3-0 … c3-17 ← searched
v2 (current=false) chunks c2-0 … c2-16 ← soft-deleted, hidden, kept for audit/rollback
v1 (purged)
"Soft-delete" means old chunks are flagged, not removed, so a rollback (bad parse of v3) is a pointer flip, not a re-ingest.
How to implement
- Assign
versionon every processed change (Tier 3). Monotonic int perdoc_id, or the source's own version number when reliable. - Chunk IDs include the version (
sha256(doc_id:version:chunk_index)), so v3 chunks never collide with v2 chunks. - Write path (ordered): write v3 chunks with
is_current=false→ verify count → set v3 chunksis_current=trueand v2 chunksis_current=false(one batch update, or a pointer in a document table that queries join on) → recorddoc.current_version=3. - Query path: mandatory filter
is_current=true(or join ondoc.current_version). This is as non-negotiable as the tenant filter. - Deletion of the document (source says gone): set
doc.deleted_at, flip all chunksis_current=false. Hard-delete after a retention window. - Garbage collection job: hard-delete chunks with
is_current=falseolder than the retention window; keep N previous versions if you need "what did the policy say in March". - Rollback: flip pointer back to v2; v3 chunks become non-current. No re-ingest.
Small example
# Postgres + pgvector sketch; same idea maps to Qdrant payload updates
# schema
"""
CREATE TABLE docs (doc_id TEXT PRIMARY KEY, tenant_id TEXT, current_version INT, deleted_at TIMESTAMPTZ);
CREATE TABLE chunks (chunk_id TEXT PRIMARY KEY, doc_id TEXT, version INT, tenant_id TEXT,
is_current BOOL DEFAULT FALSE, text TEXT, embedding VECTOR(1024), created_at TIMESTAMPTZ DEFAULT now());
CREATE INDEX ON chunks (doc_id, version);
CREATE INDEX ON chunks (tenant_id, is_current);
"""
def publish_version(conn, doc_id, tenant_id, new_version, chunk_rows):
with conn.transaction():
conn.executemany(
"INSERT INTO chunks(chunk_id, doc_id, version, tenant_id, is_current, text, embedding) "
"VALUES (%s,%s,%s,%s,FALSE,%s,%s) ON CONFLICT (chunk_id) DO NOTHING", chunk_rows)
n = conn.execute("SELECT count(*) FROM chunks WHERE doc_id=%s AND version=%s",
(doc_id, new_version)).fetchone()[0]
assert n == len(chunk_rows), "partial write"
conn.execute("UPDATE chunks SET is_current = (version = %s) WHERE doc_id=%s", (new_version, doc_id))
conn.execute("INSERT INTO docs(doc_id, tenant_id, current_version) VALUES (%s,%s,%s) "
"ON CONFLICT (doc_id) DO UPDATE SET current_version=EXCLUDED.current_version, deleted_at=NULL",
(doc_id, tenant_id, new_version))
def rollback(conn, doc_id, to_version):
with conn.transaction():
conn.execute("UPDATE chunks SET is_current = (version = %s) WHERE doc_id=%s", (to_version, doc_id))
conn.execute("UPDATE docs SET current_version=%s WHERE doc_id=%s", (to_version, doc_id))
def gc(conn, keep_versions=2, older_than_days=30):
conn.execute("""
DELETE FROM chunks c USING docs d
WHERE c.doc_id=d.doc_id AND NOT c.is_current
AND c.version < d.current_version - %s
AND c.created_at < now() - (%s || ' days')::interval""", (keep_versions - 1, older_than_days))
# query: ... WHERE tenant_id=%s AND is_current ORDER BY embedding <=> %s LIMIT 30
Pros
- Correctness: only current facts are retrievable.
- Atomic cutover: users never see a half-written document (some v3 chunks, some v2).
- Instant rollback on bad parses or bad chunking.
- Audit and time travel: "what did the policy say on date X" is answerable if you keep versions.
- Works hand-in-hand with parent–child consistency (parent–child) and embedding migrations (versioning & migration) — both are "publish a new version, flip a pointer" operations.
Cons
- Storage grows with retained versions; GC is mandatory.
- Every query needs the current-version filter; forget it once and stale answers return. Enforce in a shared retrieval function, not per call site.
- Vector stores without transactions make the pointer flip non-atomic; you need a document-level pointer that queries respect, or accept a brief window of double-counting.
- Version semantics differ per source: Confluence has real versions; a file share has none (use your own counter).
When to use
Always. Any corpus where documents change — which is every corpus except immutable archives.
When NOT to use
- Truly immutable content (published papers, closed tickets) — version is always 1, but keep the field; something will get re-parsed eventually.
Production at scale
Pointer, not scan. Don't implement "current" by deleting old chunks synchronously in the write path — deletions in ANN indexes are slow (HNSW tombstones) and a crash mid-delete leaves a mixed state. Flip a flag or a pointer; delete later.
Stores without transactions (Qdrant, Pinecone, etc.). Use the document table (Postgres) as source of truth for current_version. Two options: (a) filter on version == current_version(doc_id) via a join — only possible in pgvector/OpenSearch; (b) keep is_current in the vector payload and update it in a batch after the write, accepting a sub-second window where both versions match, then dedupe by doc_id at query time preferring the higher version. Option (b) plus a query-time "one hit per doc_id, max version" rule is robust and cheap.
Parent–child. Version parents and children together; the pointer flip covers both (parent–child).
Deleted documents are a compliance issue. When the source deletes, flip to non-current immediately on the next sync, and hard-delete within the retention SLA (often 30 days, sometimes 24 h for GDPR erasure requests). Build the erasure path as a first-class API: erase(doc_id) → all versions, all chunks, all caches, all logs referencing the text (data residency & retention).
Version explosion on noisy sources. A wiki page edited 40 times a day produces 40 versions. Debounce: only publish a new version if the content hash changed and at least N minutes passed since the last publish, or coalesce edits in the sync window.
Retention policy per source. Legal/regulatory docs: keep all versions. Chat logs / tickets: keep current only. Make it config.
HNSW deletion behaviour. Tombstoned vectors still consume memory and get visited during search until a rebuild/compaction. Schedule compaction (Qdrant optimizer, pgvector REINDEX/VACUUM) and monitor the tombstone ratio; >20% deleted vectors is a signal to rebuild the segment.
Observability. Docs with current_version but zero current chunks (broken publish); chunks with is_current=true under two different versions of the same doc (broken flip); non-current chunks past retention (GC lag). All three are nightly checks with alerts.
Migrations reuse this. A new embedding model = publish a new version of every doc under a new embedding_model_version, flip pointers when the eval passes (versioning & migration). You already built the machinery.
Prod checklist
-
doc_id + versionon every chunk; chunk IDs include version - Ordered publish: write → verify → flip → record; rollback = flip
-
is_current(or pointer join) filter enforced in the shared retrieval function - Query-time "one hit per doc_id, max version" safety net
- Soft-delete on source deletion; hard-delete/erasure job with SLA
- GC job with per-source retention config
- Debounce for high-churn sources
- ANN tombstone monitoring and compaction schedule
- Nightly consistency checks (orphans, double-current, GC lag)
Metrics to watch
| Metric | Why |
|---|---|
| Docs with 0 current chunks | Broken publish |
| Docs with >1 current version | Broken flip |
| Non-current chunks past retention | GC lag / storage |
| Tombstone ratio in ANN segments | Search speed |
| Erasure requests completed within SLA | Compliance |
Related
- Change detection — decides when a new version is created
- Parent–child Parent–child — versioned together
- Embedding versioning — same pointer-flip machinery
- Data residency & retention — erasure obligations