Evaluation → Retrieval evals (recall@k, precision@k, MRR, nDCG)
What it is
RAG has two failure sources. Retrieval evaluation isolates the first one:
| Metric | What it answers | When to use |
|---|---|---|
| recall@k | Was the answer-bearing passage in the top k? | The primary RAG metric. k = what you actually send to the LLM (and a wider k for the pre-rerank pool) |
| precision@k | What fraction of the top k was relevant? | Context efficiency — low precision wastes budget and distracts (failure modes: poisoning, distraction, confusion, clash) |
| MRR (mean reciprocal rank) | How high was the first relevant result? | Single-answer lookups |
| nDCG@k | Graded relevance, position-discounted | When relevance has degrees and order matters — the standard reranking metric |
| hit rate | Did any relevant passage appear? | Coarse, useful for dashboards |
The critical practice is measuring at two stages: recall of the retrieval pool (before reranking, k=50) and precision/nDCG of the final context (after reranking, k=5–8). They diagnose different problems:
recall@50 low → retrieval problem: chunking, embeddings, hybrid, query understanding
recall@50 high, nDCG@5 low → ranking problem: reranker
both high, answers still wrong → generation problem (generation evals)
How to implement
- Ground truth as relevant chunk IDs per query (golden dataset first), ideally at document+span level so re-chunking doesn't invalidate labels.
- Graded relevance where you can afford it: 2 = contains the answer, 1 = useful context, 0 = irrelevant. Enables nDCG; binary is acceptable for recall.
- Evaluate as the real caller: with ACL filters applied (tenant / ACL capture at ingest), because unfiltered retrieval scores are fiction in a multi-tenant system.
- Measure at both stages — pool recall and post-rerank precision/nDCG.
- Break down by query type and source (factoid, multi-hop, identifier-heavy, unanswerable) — aggregate recall hides that identifier queries are failing.
- Ablate the components: dense-only, BM25-only, hybrid, hybrid+rerank. This tells you what each part is worth on your corpus (hybrid search, reranking).
- Include unanswerable queries: for those, the correct behaviour is a low top score (score thresholding & abstention), not high recall.
Small example
import numpy as np
def recall_at_k(retrieved_ids, relevant_ids, k):
if not relevant_ids: return None
return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(relevant_ids)
def precision_at_k(retrieved_ids, relevant_ids, k):
if not retrieved_ids[:k]: return 0.0
return len(set(retrieved_ids[:k]) & set(relevant_ids)) / len(retrieved_ids[:k])
def mrr(retrieved_ids, relevant_ids):
for i, cid in enumerate(retrieved_ids, start=1):
if cid in relevant_ids: return 1.0 / i
return 0.0
def ndcg_at_k(retrieved_ids, grades: dict, k=10):
"""grades: {chunk_id: 0|1|2}"""
dcg = sum((2 ** grades.get(cid, 0) - 1) / np.log2(i + 1) for i, cid in enumerate(retrieved_ids[:k], start=1))
ideal = sorted(grades.values(), reverse=True)[:k]
idcg = sum((2 ** g - 1) / np.log2(i + 1) for i, g in enumerate(ideal, start=1))
return dcg / idcg if idcg else 0.0
def evaluate_retrieval(cases, system, pool_k=50, final_k=6):
rows = []
for c in cases:
caller = load_caller(c.caller) # ACL-filtered: the real conditions
pool = system.retrieve_pool(c.query, caller, k=pool_k) # pre-rerank
final = system.rerank(c.query, pool)[:final_k] # post-rerank
pool_ids = [h["chunk_id"] for h in pool]
final_ids = [h["chunk_id"] for h in final]
rows.append({
"case_id": c.case_id, "query_type": c.query_type,
"recall@50": recall_at_k(pool_ids, c.relevant_chunk_ids, pool_k),
"recall@6": recall_at_k(final_ids, c.relevant_chunk_ids, final_k),
"precision@6": precision_at_k(final_ids, c.relevant_chunk_ids, final_k),
"mrr": mrr(final_ids, c.relevant_chunk_ids),
"ndcg@6": ndcg_at_k(final_ids, c.grades or {i: 2 for i in c.relevant_chunk_ids}, final_k),
"top_score": final[0]["rerank_score"] if final else 0.0,
"unanswerable": c.query_type == "unanswerable",
})
return rows
def ablation(cases, system):
"""What is each component worth on THIS corpus?"""
configs = {
"dense_only": lambda q, c: system.dense(q, c, k=50),
"bm25_only": lambda q, c: system.bm25(q, c, k=50),
"hybrid": lambda q, c: system.hybrid(q, c, k=50),
"hybrid_rerank": lambda q, c: system.rerank(q, system.hybrid(q, c, k=50))[:6],
}
out = {}
for name, fn in configs.items():
scores = [recall_at_k([h["chunk_id"] for h in fn(c.query, load_caller(c.caller))],
c.relevant_chunk_ids, 50)
for c in cases if c.relevant_chunk_ids]
out[name] = {"recall": round(np.mean([s for s in scores if s is not None]), 3)}
return out
def diagnose(rows):
"""The decision tree that tells you where to spend effort."""
ans = [r for r in rows if not r["unanswerable"]]
pool_recall = np.mean([r["recall@50"] for r in ans if r["recall@50"] is not None])
final_ndcg = np.mean([r["ndcg@6"] for r in ans])
if pool_recall < 0.85:
return f"RETRIEVAL: pool recall {pool_recall:.2f} — fix chunking / embeddings / hybrid / query understanding"
if final_ndcg < 0.7:
return f"RANKING: pool recall {pool_recall:.2f} but nDCG@6 {final_ndcg:.2f} — fix the reranker"
return "Retrieval healthy — investigate generation (generation evals)"
Pros
- Localises the problem — the single most useful diagnostic in a RAG system.
- Fast and cheap: no LLM calls needed for recall/precision/nDCG.
- Ablations quantify each component's value on your corpus, so you can justify or drop hybrid, reranking, contextual retrieval, and so on.
- Deterministic, so small changes are detectable without judge noise.
- Directly gates chunking, embedding, and index changes (chunking, embeddings, indexing & vector stores).
Cons
- Labelling relevant chunks is expensive, and labels break when chunking changes (mitigate with span-level ground truth).
- Incomplete labels understate performance: a retrieved passage that answers the question but wasn't labelled counts as a miss. Pooling (label the union of what several systems retrieve) mitigates this.
- Binary relevance is crude for nDCG; graded labels cost more.
- Retrieval metrics don't capture whether the combination of passages supports a complete answer — that's generation-side (generation evals).
When to use
Always, and first. Before tuning prompts, before changing models, before adding advanced RAG patterns. The diagnosis "is the answer even in the context" determines everything downstream.
When NOT to use (alone)
- As the only measure of system quality — high recall with bad generation is still a bad product (component vs end-to-end).
- On unanswerable queries, where recall is undefined; measure top-score distribution and abstention instead (score thresholding & abstention).
Production at scale
Span-level ground truth survives re-chunking. Label the document and character range that contains the answer; at eval time, resolve to whichever chunk IDs currently cover that span. This is the difference between a golden set that lasts a year and one that breaks on your next chunking change (chunking).
Pooled labelling for completeness. Run several retrieval configurations, take the union of their top-20, and have annotators label that pool. This corrects the bias where your current system defines what "relevant" means.
Always evaluate with ACL filters on. Unfiltered recall in a multi-tenant system is a fantasy number; filtered ANN behaves differently (filtered search), and it's the filtered number that matters.
Two-stage reporting as standard: pool recall and final nDCG on every run, plus the diagnosis line. Teams that report only one number spend weeks tuning the wrong component.
Per-query-type breakdown is mandatory. Identifier-heavy queries failing while conceptual queries pass is the signature of missing or broken hybrid search (hybrid search) — invisible in an aggregate.
Track the pre-rerank pool size as a tuning parameter: sweep k ∈ {20, 50, 100} against pool recall and end-to-end quality to find the plateau (reranking).
Corpus drift: rerun retrieval evals after significant ingestion changes. Recall can fall simply because the corpus grew and the answer now competes with more near-neighbours.
Cheap enough to run continuously: retrieval evals need no LLM calls, so run them nightly against production indexes (read-only, in a staging namespace) as a canary (index types: Flat, HNSW, IVF-PQ).
Prod checklist
- Relevant passages labelled at document+span level, resolved to chunk IDs at eval time
- Graded relevance (0/1/2) where nDCG matters; pooled labelling across configurations
- Evaluation runs with real ACL filters and realistic callers
- Two-stage measurement: pool recall (pre-rerank) and precision/nDCG (post-rerank)
- Per-query-type and per-source breakdowns
- Component ablation (dense / BM25 / hybrid / +rerank) recorded
- Unanswerable queries measured by top-score distribution, not recall
- Pool-size sweep documented
- Nightly retrieval canary against the production index
Metrics to watch
| Metric | Why |
|---|---|
| recall@pool_k | The gating number — was the answer even available |
| nDCG@final_k | Ranking quality |
| Per-query-type recall (esp. identifier queries) | Hybrid search health |
| Ablation deltas | What each component is worth |
| Recall trend after ingestion changes | Corpus drift |
Related
- Golden dataset — the labels
- Generation evals — the other half
- Component vs end-to-end — the diagnostic split
- Hybrid, Reranking — what ablations measure
- Index types — ANN recall canary