Embeddings → Versioning & migration (dual-index cutover)

Embedding Model Migration for RAG: Dual-Index Cutover Explained

What it is

An embedding space is defined by the model that produced it. Query vectors from model B searched against document vectors from model A return garbage — not slightly worse, garbage. So any of these events is a migration:

The safe pattern is dual-index:

t0: index_A (live, version A)
t1: build index_B (version B) from the same chunk registry, in the background, rate-limited
t2: evaluate B vs A on the golden set (retrieval evals) and shadow traffic
t3: cut over: query service reads B; ingestion writes to B (and A until confirmed)
t4: delete A after a rollback window

How to implement

  1. embedding_model_version on every chunk (metadata on every chunk) and on the index/collection itself; the query service asserts its embedder version equals the index version.
  2. Chunk registry independent of the vector store (change detection): chunk IDs, text, metadata. A re-index re-embeds from the registry; it never re-fetches or re-parses (unless the migration is a parser change).
  3. Build the new index as a separate collection/namespace (chunks_v2), from a background job: paginate the registry, embed in batches, upsert; checkpointed and resumable (state & durability); rate-limited to protect ingestion and serving.
  4. Keep both indexes current during the build: new/changed documents are written to A and B (dual-write) so B isn't stale at cutover.
  5. Evaluate B: golden-set recall/nDCG; end-to-end answer quality; latency; shadow a slice of live queries to both and compare.
  6. Cut over with a config flag (per tenant if you need staged rollout); monitor; keep A for a rollback window.
  7. Roll back = flip the flag. Finish = stop dual-write, delete A, reclaim memory.

Small example

# config
INDEX = {"live": "chunks_v1", "candidate": "chunks_v2", "dual_write": True}
EMBED_VERSION = {"chunks_v1": "e5ml-large|q=query:|d=passage:", "chunks_v2": "e5-acme-ft-v1|q=query:|d=passage:"}

def query_service_startup():
    assert embedder.version == EMBED_VERSION[INDEX["live"]], "query embedder ≠ live index version"

def ingest_write(chunk, text):
    vec_a = embed_documents_v1([text])[0]; store.upsert(INDEX["live"], chunk.id, vec_a, chunk.payload | {"emb_v": EMBED_VERSION["chunks_v1"]})
    if INDEX["dual_write"]:
        vec_b = embed_documents_v2([text])[0]; store.upsert(INDEX["candidate"], chunk.id, vec_b, chunk.payload | {"emb_v": EMBED_VERSION["chunks_v2"]})

def reindex_job(registry, target="chunks_v2", batch=256, checkpoint_key="reindex:v2:cursor", rate_per_sec=2000):
    cursor = kv.get(checkpoint_key)
    for page in registry.iter_current_chunks(after=cursor, page_size=batch):   # only is_current chunks
        vecs = embed_documents_v2([c.text_indexed for c in page])
        store.upsert_many(target, [(c.id, v, c.payload | {"emb_v": EMBED_VERSION[target]}) for c, v in zip(page, vecs)])
        kv.set(checkpoint_key, page[-1].id)
        throttle(rate_per_sec, len(page))
    kv.set("reindex:v2:status", "complete")

def cutover():
    assert kv.get("reindex:v2:status") == "complete"
    assert golden_recall("chunks_v2") >= golden_recall("chunks_v1") - 0.005
    INDEX["live"], INDEX["candidate"] = "chunks_v2", "chunks_v1"      # flag flip; A kept for rollback window

Pros

  • Zero-downtime migrations with a measured go/no-go and instant rollback.
  • No mixed-version index — the failure mode where half the corpus is unsearchable.
  • Version fields make it possible to audit what's live and to find stragglers.
  • Same machinery serves chunking/parser/contextual migrations.

Cons

  • Double memory/storage during the migration window (plan capacity).
  • Double embedding cost on dual-write, plus the full re-embed of the corpus.
  • Long builds on large corpora (hours to days) — the registry must stay consistent and dual-write must not be skipped.
  • Managed vector stores may bill by index count/size — the migration window costs money.

When to use

Every embedding-affecting change. There is no safe in-place migration.

When NOT to use (dual-index)

  • Tiny corpora (thousands of chunks) where a full rebuild takes minutes and a short read-only window is acceptable — still keep the version field and the eval gate.

Production at scale

Capacity planning. Two full indexes for the window: memory (HNSW is RAM-resident), storage, and GPU/API time for the re-embed. Schedule builds off-peak; rate-limit to protect serving p95.

Resumability. Checkpoint by registry cursor; upserts are idempotent (same chunk ID). A crashed job resumes, not restarts. Track progress % on a dashboard.

Consistency during the build. Dual-write starts before the build starts (or the job re-scans changes since its start cursor at the end). Otherwise documents updated mid-build are stale in B.

Only current chunks. Re-embed is_current=true only (document versioning); old versions don't need to migrate unless you serve time-travel queries.

Staged cutover per tenant. Flip the flag for internal tenants → 5% → 50% → 100%; compare online metrics (online eval) at each stage; each stage is a rollback point.

Sparse/BM25 indexes (hybrid search) are unaffected by embedding changes but are affected by indexed-text changes (contextual retrieval, breadcrumbs). Include them in the migration plan when text changes.

Stragglers audit: after cutover, count chunks in the live index by emb_v; anything ≠ live version is a bug. Also verify every current registry chunk exists in the live index (no drops).

Cost accounting: $ per migration = re-embed tokens + dual-write overhead + double index for N days. Put it in the decision to migrate (model choice: don't churn models).

Runbook: written, rehearsed, with owners and a rollback command. Deprecation notices from API providers turn this from optional into urgent.

Prod checklist

  • embedding_model_version on chunks and index; query service asserts equality
  • Chunk registry independent of the vector store; re-index reads from it
  • Background build: checkpointed, resumable, rate-limited, current-chunks only
  • Dual-write from before build start; late-change re-scan
  • Eval gate (golden + shadow traffic) before cutover
  • Flag-based cutover, staged per tenant; rollback window; then delete old
  • Straggler and completeness audits post-cutover
  • BM25/sparse indexes included when indexed text changes
  • Capacity plan and cost estimate per migration; runbook rehearsed

Metrics to watch

MetricWhy
Re-index progress %, throughput, ETAOps
Golden recall A vs B; shadow-traffic agreementGo/no-go
Serving p95 during buildRate-limit adequacy
Chunks by emb_v in live index post-cutoverStragglers
Registry vs index countCompleteness