Evaluation → Golden dataset first
What it is
The golden dataset is the ground truth your system is measured against. Not MMLU, not MTEB, not a vendor's benchmark — your questions, your documents, your correct answers.
A minimal RAG golden set has, per case:
{
"case_id": "GOLD-142",
"query": "How many days notice do we need to give to terminate the Acme MSA?",
"query_type": "factoid", // factoid | multi_hop | aggregate | unanswerable | ambiguous
"caller": {"tenant": "t_acme", "groups": ["legal"]},
"relevant_chunk_ids": ["c_8812", "c_8813"], // retrieval ground truth
"expected_answer": "30 days' written notice",
"expected_citations": ["c_8812"],
"must_include": ["30 days", "written"],
"must_not_include": ["60 days"],
"expected_behaviour": "answer", // answer | abstain | clarify | partial
"source": "production_failure", // seed | production_failure | incident | manual
"added": "2024-06-03",
"notes": "Model previously read the 2022 version"
}
Four things most teams get wrong:
- Too small to detect anything. 20 cases means a 5-point move is 1 case. 50–200 is the working range; more for high-stakes systems.
- No unanswerable cases. Without questions the corpus genuinely cannot answer, you cannot measure correct abstention or false refusal (score thresholding & abstention) — the most-skipped RAG metrics (metrics cheat sheet).
- Synthetic questions only. LLM-generated questions are stylistically uniform and miss how users actually ask. They're fine for bootstrapping, not for judging.
- Never grown. A static golden set stops representing production within months.
How to implement
- Seed from real queries. Pull 100–200 from logs (or from the people who will use the system, before launch). Stratify by query type, source, tenant, and difficulty.
- Label with domain experts, not engineers guessing. For retrieval ground truth, label the chunks that contain the answer; for generation, write the expected answer and the must-include facts.
- Include the hard categories deliberately: unanswerable (~15–20%), multi-hop, aggregate/numeric, conflicting-sources, permission-scoped (the same question for two callers with different access), and ambiguous.
- Version it as data in git/object storage with a manifest; tie every eval result to a golden-set version.
- Freeze a test split you don't look at during development, to catch overfitting to the dev set.
- Grow from failures: every production incident, thumbs-down with a clear cause, and support escalation becomes a case, with the fix's regression test.
- Automate the run: one command produces retrieval metrics, generation metrics, and a diff against the last version (the testing pyramid).
Small example
from pydantic import BaseModel
from typing import Literal
class GoldenCase(BaseModel):
case_id: str
query: str
query_type: Literal["factoid", "multi_hop", "aggregate", "unanswerable", "ambiguous", "conflicting", "permission"]
caller: dict
relevant_chunk_ids: list[str] = []
expected_answer: str | None = None
expected_citations: list[str] = []
must_include: list[str] = []
must_not_include: list[str] = []
expected_behaviour: Literal["answer", "abstain", "clarify", "partial"] = "answer"
source: str
added: str
corpus_version: str # which corpus snapshot this was labelled against
def run_golden(cases: list[GoldenCase], system, judge) -> dict:
rows = []
for c in cases:
caller = load_caller(c.caller)
retrieved = system.retrieve(c.query, caller, k=20)
result = system.answer(c.query, caller)
retrieved_ids = [r["chunk_id"] for r in retrieved]
recall = (len(set(c.relevant_chunk_ids) & set(retrieved_ids[:10])) / len(c.relevant_chunk_ids)
if c.relevant_chunk_ids else None)
abstained = result.get("refused", False)
rows.append({
"case_id": c.case_id, "query_type": c.query_type,
"recall@10": recall,
"behaviour_correct": (abstained if c.expected_behaviour == "abstain" else not abstained),
"must_include": all(s.lower() in (result.get("answer") or "").lower() for s in c.must_include),
"must_not_include": not any(s.lower() in (result.get("answer") or "").lower() for s in c.must_not_include),
"citation_valid": set(result.get("citations", [])).issubset(set(retrieved_ids)),
"judge_score": judge(c, result) if c.expected_answer and not abstained else None,
"cost_usd": result.get("cost_usd"), "latency_ms": result.get("latency_ms"),
})
return summarise(rows) # per query_type and overall; never just the mean
def summarise(rows):
import pandas as pd
df = pd.DataFrame(rows)
overall = {"n": len(df), "recall@10": df["recall@10"].mean(),
"behaviour_correct": df["behaviour_correct"].mean(),
"must_include_pass": df["must_include"].mean(),
"citation_valid": df["citation_valid"].mean(),
"judge_mean": df["judge_score"].mean(),
"p95_latency_ms": df["latency_ms"].quantile(0.95),
"cost_per_query": df["cost_usd"].mean()}
by_type = df.groupby("query_type").agg({"recall@10": "mean", "behaviour_correct": "mean",
"judge_score": "mean"}).to_dict("index")
return {"overall": overall, "by_query_type": by_type} # breakdown is where the signal is
Pros
- The only way to know whether a change helped. Every prompt, chunking, model, and retrieval decision in this reference is settled by running this.
- Cheap relative to its value: a few person-days to build, then it pays forever.
- Turns incidents into permanent protection — the same bug cannot ship twice.
- Enables CI gating (the testing pyramid) so regressions are caught before users see them.
- Per-category breakdowns localise problems (retrieval vs generation vs abstention).
Cons
- Labelling is real work, and needs domain experts whose time is scarce.
- Goes stale: as the corpus changes, ground-truth chunk IDs drift (a re-chunk invalidates them — store
corpus_versionand re-label or map). - Small sets have wide confidence intervals — a 3-point move on 50 cases means nothing (statistical hygiene).
- Overfitting risk if you tune against the same set repeatedly without a held-out split.
- Doesn't cover everything: rare tail cases and adversarial inputs need separate suites (the testing pyramid).
When to use
Before anything else. The first artifact of an AI system is not the retriever or the prompt; it's the golden set. Building it first also forces you to define what "correct" means, which is usually the hardest unresolved question in the project.
When NOT to use
- Never skip it. But for a two-day prototype, 20 cases you wrote yourself is enough to start — just don't report percentages from it.
Production at scale
Stratified, not random. Sample by query type, source, tenant size, and language so that per-category metrics are meaningful. A set that's 90% factoid tells you nothing about multi-hop performance.
Corpus versioning is the maintenance burden. Ground-truth chunk IDs break when chunking config or parsers change (chunking, OCR / digitization). Two mitigations: label at document + span level and resolve to chunk IDs at eval time, or re-map IDs during migrations (versioning & migration). The span-level approach survives re-chunking and is worth the extra labelling effort.
Permission cases are mandatory in multi-tenant systems: the same question asked by two callers with different access, with different expected answers (authentication & authorization). These catch the leaks that no other test does.
Grow continuously, prune deliberately. Add from failures weekly (data flywheel); retire cases whose documents no longer exist. Target a steady size rather than unbounded growth — 200 well-chosen cases beat 2,000 stale ones.
Held-out split: 70% dev (used constantly), 30% test (run before releases only). Divergence between them is your overfitting signal.
Human agreement: double-label 10% and measure agreement. If experts disagree on the correct answer, your task definition is ambiguous and no model will satisfy it.
Cost and latency belong in the eval output, not just quality. A change that adds 4 points of accuracy and 3× the cost is a decision, not an automatic win (agent evals).
Make it one command. make eval producing a comparison against the previous run is the difference between a golden set that gets used and one that rots in a folder.
Prod checklist
- 50–200+ real cases, stratified by query type, source, tenant, language
- 15–20% unanswerable cases; multi-hop, aggregate, conflicting, and permission cases included
- Labelled by domain experts; 10% double-labelled with agreement measured
- Ground truth at document+span level, resolved to chunk IDs at eval time
- Versioned dataset with
corpus_version; results tied to a set version - Held-out test split not used during development
- One-command run producing per-category breakdowns plus cost and latency
- Weekly growth from production failures; stale cases retired
- Wired into CI as a regression gate (the testing pyramid)
Metrics to watch
| Metric | Why |
|---|---|
| Per-query-type scores (not just the mean) | Where the system actually fails |
| Dev vs held-out test divergence | Overfitting |
| Inter-annotator agreement | Task definition clarity |
| Golden-set size, age, and growth rate | Representativeness |
| Cases traceable to production incidents | Flywheel health |
Related
- Prompt evaluation, Retrieval evals, Generation evals — what runs against it
- Statistical hygiene — how to read the numbers
- Testing pyramid — CI gating
- Data flywheel — how it grows
- Abstention — why unanswerable cases matter