Embeddings → Versioning & migration (dual-index cutover)
What it is
An embedding space is defined by the model that produced it. Query vectors from model B searched against document vectors from model A return garbage — not slightly worse, garbage. So any of these events is a migration:
- New embedding model (model choice) or fine-tuned version (fine-tuning embeddings on your own logs)
- Prefix/instruction change (symmetric vs asymmetric)
- Dimension change / Matryoshka truncation policy change
- Late chunking toggled (late chunking)
- Quantisation change that alters stored vectors
- Also (same machinery): chunking config change (chunking), parser upgrade (OCR / digitization), contextual-retrieval model change (contextual retrieval) — anything that changes indexed text
The safe pattern is dual-index:
t0: index_A (live, version A)
t1: build index_B (version B) from the same chunk registry, in the background, rate-limited
t2: evaluate B vs A on the golden set (retrieval evals) and shadow traffic
t3: cut over: query service reads B; ingestion writes to B (and A until confirmed)
t4: delete A after a rollback window
How to implement
embedding_model_versionon every chunk (metadata on every chunk) and on the index/collection itself; the query service asserts its embedder version equals the index version.- Chunk registry independent of the vector store (change detection): chunk IDs, text, metadata. A re-index re-embeds from the registry; it never re-fetches or re-parses (unless the migration is a parser change).
- Build the new index as a separate collection/namespace (
chunks_v2), from a background job: paginate the registry, embed in batches, upsert; checkpointed and resumable (state & durability); rate-limited to protect ingestion and serving. - Keep both indexes current during the build: new/changed documents are written to A and B (dual-write) so B isn't stale at cutover.
- Evaluate B: golden-set recall/nDCG; end-to-end answer quality; latency; shadow a slice of live queries to both and compare.
- Cut over with a config flag (per tenant if you need staged rollout); monitor; keep A for a rollback window.
- Roll back = flip the flag. Finish = stop dual-write, delete A, reclaim memory.
Small example
# config
INDEX = {"live": "chunks_v1", "candidate": "chunks_v2", "dual_write": True}
EMBED_VERSION = {"chunks_v1": "e5ml-large|q=query:|d=passage:", "chunks_v2": "e5-acme-ft-v1|q=query:|d=passage:"}
def query_service_startup():
assert embedder.version == EMBED_VERSION[INDEX["live"]], "query embedder ≠ live index version"
def ingest_write(chunk, text):
vec_a = embed_documents_v1([text])[0]; store.upsert(INDEX["live"], chunk.id, vec_a, chunk.payload | {"emb_v": EMBED_VERSION["chunks_v1"]})
if INDEX["dual_write"]:
vec_b = embed_documents_v2([text])[0]; store.upsert(INDEX["candidate"], chunk.id, vec_b, chunk.payload | {"emb_v": EMBED_VERSION["chunks_v2"]})
def reindex_job(registry, target="chunks_v2", batch=256, checkpoint_key="reindex:v2:cursor", rate_per_sec=2000):
cursor = kv.get(checkpoint_key)
for page in registry.iter_current_chunks(after=cursor, page_size=batch): # only is_current chunks
vecs = embed_documents_v2([c.text_indexed for c in page])
store.upsert_many(target, [(c.id, v, c.payload | {"emb_v": EMBED_VERSION[target]}) for c, v in zip(page, vecs)])
kv.set(checkpoint_key, page[-1].id)
throttle(rate_per_sec, len(page))
kv.set("reindex:v2:status", "complete")
def cutover():
assert kv.get("reindex:v2:status") == "complete"
assert golden_recall("chunks_v2") >= golden_recall("chunks_v1") - 0.005
INDEX["live"], INDEX["candidate"] = "chunks_v2", "chunks_v1" # flag flip; A kept for rollback window
Pros
- Zero-downtime migrations with a measured go/no-go and instant rollback.
- No mixed-version index — the failure mode where half the corpus is unsearchable.
- Version fields make it possible to audit what's live and to find stragglers.
- Same machinery serves chunking/parser/contextual migrations.
Cons
- Double memory/storage during the migration window (plan capacity).
- Double embedding cost on dual-write, plus the full re-embed of the corpus.
- Long builds on large corpora (hours to days) — the registry must stay consistent and dual-write must not be skipped.
- Managed vector stores may bill by index count/size — the migration window costs money.
When to use
Every embedding-affecting change. There is no safe in-place migration.
When NOT to use (dual-index)
- Tiny corpora (thousands of chunks) where a full rebuild takes minutes and a short read-only window is acceptable — still keep the version field and the eval gate.
Production at scale
Capacity planning. Two full indexes for the window: memory (HNSW is RAM-resident), storage, and GPU/API time for the re-embed. Schedule builds off-peak; rate-limit to protect serving p95.
Resumability. Checkpoint by registry cursor; upserts are idempotent (same chunk ID). A crashed job resumes, not restarts. Track progress % on a dashboard.
Consistency during the build. Dual-write starts before the build starts (or the job re-scans changes since its start cursor at the end). Otherwise documents updated mid-build are stale in B.
Only current chunks. Re-embed is_current=true only (document versioning); old versions don't need to migrate unless you serve time-travel queries.
Staged cutover per tenant. Flip the flag for internal tenants → 5% → 50% → 100%; compare online metrics (online eval) at each stage; each stage is a rollback point.
Sparse/BM25 indexes (hybrid search) are unaffected by embedding changes but are affected by indexed-text changes (contextual retrieval, breadcrumbs). Include them in the migration plan when text changes.
Stragglers audit: after cutover, count chunks in the live index by emb_v; anything ≠ live version is a bug. Also verify every current registry chunk exists in the live index (no drops).
Cost accounting: $ per migration = re-embed tokens + dual-write overhead + double index for N days. Put it in the decision to migrate (model choice: don't churn models).
Runbook: written, rehearsed, with owners and a rollback command. Deprecation notices from API providers turn this from optional into urgent.
Prod checklist
-
embedding_model_versionon chunks and index; query service asserts equality - Chunk registry independent of the vector store; re-index reads from it
- Background build: checkpointed, resumable, rate-limited, current-chunks only
- Dual-write from before build start; late-change re-scan
- Eval gate (golden + shadow traffic) before cutover
- Flag-based cutover, staged per tenant; rollback window; then delete old
- Straggler and completeness audits post-cutover
- BM25/sparse indexes included when indexed text changes
- Capacity plan and cost estimate per migration; runbook rehearsed
Metrics to watch
| Metric | Why |
|---|---|
| Re-index progress %, throughput, ETA | Ops |
| Golden recall A vs B; shadow-traffic agreement | Go/no-go |
| Serving p95 during build | Rate-limit adequacy |
Chunks by emb_v in live index post-cutover | Stragglers |
| Registry vs index count | Completeness |
Related
- Chunk metadata — version fields
- Change detection — config-driven reprocessing waves
- Document versioning — same pointer-flip idea
- Model choice / fine-tuning embeddings on your own logs — what triggers migrations
- Versioning & rollout