Embeddings → Fine-tuning embeddings on your own logs

Fine-Tuning Embedding Models for RAG: Train on Your Own Logs

What it is

General embedding models learn similarity from web-scale data. Your corpus has its own vocabulary ("MSA", "SKU-4471", "Tier 2 escalation"), its own notion of relevance (a clause is relevant to a question about notice periods even if it never uses the word "notice"), and its own hard negatives (the 2022 policy vs the 2024 policy). Fine-tuning adapts the model with contrastive learning:

loss pulls  q ──► p+   (relevant passage) together
loss pushes q ──► p−   (hard negative: looks relevant, isn't) apart
plus in-batch negatives (other examples' passages)

Training data sources, in order of value:

  1. Golden set (retrieval evals): human-labelled query → relevant chunk IDs.
  2. Production clicks/citations: queries where the user clicked/used a result, or where the generated answer cited a chunk and got a thumbs-up.
  3. Synthetic: an LLM generates plausible questions for each chunk (doc → questions). Cheap, large, lower quality; good for bootstrapping.
  4. Hard negatives: for each query, top-ranked passages from the current retriever that are not relevant (BM25 or dense top-k minus positives), or a reranker's low-scored high-similarity hits. Hard negatives are what make the fine-tune sharp; random negatives teach nothing.

Also applies to rerankers (cross-encoders) — same data, often an even bigger gain (reranking).

How to implement

  1. Assemble triplets: (query, positive_chunk_id, [hard_negative_ids]). Deduplicate; strip PII (PII); hold out a test split by query from day one; make sure test queries and their documents aren't in training.
  2. Mine hard negatives with the current retriever: top-20 by dense/BM25 excluding positives; optionally filter false negatives with a reranker/LLM judge (a "negative" that actually answers the query poisons training).
  3. Train with sentence-transformers (MultipleNegativesRankingLoss or CachedMNRL for big batches), starting from your production model; a few epochs; learning rate ~1e-5–2e-5; batch size as large as memory allows (in-batch negatives matter).
  4. Evaluate on the held-out golden queries: recall@k, nDCG vs the base model (retrieval evals). Also check for regressions on general queries if you serve them.
  5. Register the new model version; re-index (versioning & migration) — a fine-tuned model produces different vectors; there is no partial migration.
  6. Repeat on a cadence as logs accumulate (data flywheel); track gains per iteration.

Small example

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader
import random, json

base = SentenceTransformer("intfloat/multilingual-e5-large")

# triplets: [{"query": ..., "pos": chunk_text, "negs": [chunk_text, ...]}]
triplets = [json.loads(l) for l in open("data/triplets_train.jsonl")]

examples = []
for t in triplets:
    neg = random.choice(t["negs"]) if t["negs"] else None
    texts = ["query: " + t["query"], "passage: " + t["pos"]] + (["passage: " + neg] if neg else [])
    examples.append(InputExample(texts=texts))            # (anchor, positive[, hard negative])

loader = DataLoader(examples, shuffle=True, batch_size=64)
loss = losses.MultipleNegativesRankingLoss(base)           # in-batch negatives + explicit hard negatives

base.fit(train_objectives=[(loader, loss)], epochs=2, warmup_steps=100,
         optimizer_params={"lr": 2e-5}, show_progress_bar=True, checkpoint_path="ckpt/", checkpoint_save_steps=500)
base.save("models/e5-large-acme-v1")

# hard-negative mining sketch (before training)
def mine_hard_negatives(query, positives, retriever, reranker, k=20):
    cands = [c for c in retriever.search(query, k) if c.id not in positives]
    scored = reranker.score(query, [c.text for c in cands])
    return [c.id for c, s in zip(cands, scored) if s < 0.3][:5]   # similar-looking but judged irrelevant

Evaluate before/after: recall_at_k(embed_queries(test_queries), embed_documents(corpus), golden, k=10) with both models.

Pros

  • Large recall gains on domain corpora (10–30 points is common when the base model was generic).
  • Cheap: hours on one GPU; no LLM retraining.
  • Compounds with logs: more usage → better retriever (data flywheel).
  • Attacks the top failure mode in RAG directly.

Cons

  • Full re-index per new model version.
  • Data quality risks: false negatives in training, PII in queries, tenant data mixing.
  • Overfitting to the query distribution you trained on; generic or new-topic queries may regress — evaluate broadly.
  • MLOps: dataset versioning, reproducibility, model registry, rollback (fine-tuning ops).
  • Fine-tuned models are tied to a base; base deprecation means retraining.

When to use

  • Retrieval recall is the bottleneck (component vs end-to-end localisation) and hybrid + rerank + chunking improvements have plateaued.
  • You have ≥ ~1k quality (query, positive) pairs — or can synthesise and then refine with real logs.
  • Domain vocabulary is far from web text.

When NOT to use

  • Before hybrid search (hybrid search) and reranking (reranking) — those are cheaper and often bigger.
  • Tiny or rapidly changing corpora where the effort isn't repaid.
  • When you can't build a held-out eval — you'd be flying blind.
  • Multi-tenant setups where training data can't be pooled and per-tenant models aren't feasible — fine-tune the reranker instead (query-time, no re-index) or use a shared scrubbed pool.

Production at scale

Data pipeline as a product. Continuous extraction of (query, cited/clicked chunk, outcome) from traces (observability & tracing) into a labelled store; weekly hard-negative mining; PII scrubbing; tenant tagging; dedupe. This is the flywheel's retrieval arm (data flywheel).

False-negative filtering matters. Hard negatives mined from the current retriever include passages that are actually relevant but unlabelled. Filter with a strong reranker or an LLM judge (LLM-as-judge); otherwise you train the model to push away correct answers.

Held-out discipline. Split by query and avoid document leakage (a test query's positive document shouldn't be the positive of a training query). Report recall on held-out golden + a general-query sanity set.

Rerankers first for multi-tenant. A fine-tuned cross-encoder improves ranking at query time with no re-index and can be swapped per tenant cheaply; fine-tuned embedders need a full re-index per version and per tenant if tenant-specific.

Migration: new embedder = new embedding_model_version; dual-index; A/B on the golden set and online; cut over; delete old (versioning & migration). Budget the re-embed cost each iteration.

Cadence: retrain when new labelled data ≥ ~20% of the previous set or when a new domain appears; not on a calendar for its own sake.

Registry and rollback: model artifacts versioned with the dataset hash and training config; rollback = point the query service and a re-index job at the previous version (fine-tuning ops).

Synthetic data caution: LLM-generated questions are stylistically uniform; mix with real queries; watch for the model learning the generator's phrasing.

Prod checklist

  • Continuous (query, positive, outcome) extraction from traces; PII scrubbed; tenant-tagged
  • Hard-negative mining with false-negative filtering
  • Held-out split by query with no document leakage; general-query sanity set
  • Training reproducible (dataset hash, config, seed); artifacts in a registry
  • Recall/nDCG gains reported vs base on held-out; regression check on general queries
  • Re-index via dual-index migration; rollback path
  • Reranker fine-tuning considered first for multi-tenant / re-index-averse setups

Metrics to watch

MetricWhy
Held-out recall@k / nDCG vs baseThe gain
General-query recallOverfitting
Labelled pairs accumulated per weekFlywheel health
Re-index cost per model iterationCadence economics