Evaluation → Golden dataset first

Golden Dataset for LLM Evaluation: Build It First

What it is

The golden dataset is the ground truth your system is measured against. Not MMLU, not MTEB, not a vendor's benchmark — your questions, your documents, your correct answers.

A minimal RAG golden set has, per case:

{
  "case_id": "GOLD-142",
  "query": "How many days notice do we need to give to terminate the Acme MSA?",
  "query_type": "factoid",            // factoid | multi_hop | aggregate | unanswerable | ambiguous
  "caller": {"tenant": "t_acme", "groups": ["legal"]},
  "relevant_chunk_ids": ["c_8812", "c_8813"],       // retrieval ground truth
  "expected_answer": "30 days' written notice",
  "expected_citations": ["c_8812"],
  "must_include": ["30 days", "written"],
  "must_not_include": ["60 days"],
  "expected_behaviour": "answer",     // answer | abstain | clarify | partial
  "source": "production_failure",     // seed | production_failure | incident | manual
  "added": "2024-06-03",
  "notes": "Model previously read the 2022 version"
}

Four things most teams get wrong:

  1. Too small to detect anything. 20 cases means a 5-point move is 1 case. 50–200 is the working range; more for high-stakes systems.
  2. No unanswerable cases. Without questions the corpus genuinely cannot answer, you cannot measure correct abstention or false refusal (score thresholding & abstention) — the most-skipped RAG metrics (metrics cheat sheet).
  3. Synthetic questions only. LLM-generated questions are stylistically uniform and miss how users actually ask. They're fine for bootstrapping, not for judging.
  4. Never grown. A static golden set stops representing production within months.

How to implement

  1. Seed from real queries. Pull 100–200 from logs (or from the people who will use the system, before launch). Stratify by query type, source, tenant, and difficulty.
  2. Label with domain experts, not engineers guessing. For retrieval ground truth, label the chunks that contain the answer; for generation, write the expected answer and the must-include facts.
  3. Include the hard categories deliberately: unanswerable (~15–20%), multi-hop, aggregate/numeric, conflicting-sources, permission-scoped (the same question for two callers with different access), and ambiguous.
  4. Version it as data in git/object storage with a manifest; tie every eval result to a golden-set version.
  5. Freeze a test split you don't look at during development, to catch overfitting to the dev set.
  6. Grow from failures: every production incident, thumbs-down with a clear cause, and support escalation becomes a case, with the fix's regression test.
  7. Automate the run: one command produces retrieval metrics, generation metrics, and a diff against the last version (the testing pyramid).

Small example

from pydantic import BaseModel
from typing import Literal

class GoldenCase(BaseModel):
    case_id: str
    query: str
    query_type: Literal["factoid", "multi_hop", "aggregate", "unanswerable", "ambiguous", "conflicting", "permission"]
    caller: dict
    relevant_chunk_ids: list[str] = []
    expected_answer: str | None = None
    expected_citations: list[str] = []
    must_include: list[str] = []
    must_not_include: list[str] = []
    expected_behaviour: Literal["answer", "abstain", "clarify", "partial"] = "answer"
    source: str
    added: str
    corpus_version: str                     # which corpus snapshot this was labelled against

def run_golden(cases: list[GoldenCase], system, judge) -> dict:
    rows = []
    for c in cases:
        caller = load_caller(c.caller)
        retrieved = system.retrieve(c.query, caller, k=20)
        result = system.answer(c.query, caller)

        retrieved_ids = [r["chunk_id"] for r in retrieved]
        recall = (len(set(c.relevant_chunk_ids) & set(retrieved_ids[:10])) / len(c.relevant_chunk_ids)
                  if c.relevant_chunk_ids else None)
        abstained = result.get("refused", False)

        rows.append({
            "case_id": c.case_id, "query_type": c.query_type,
            "recall@10": recall,
            "behaviour_correct": (abstained if c.expected_behaviour == "abstain" else not abstained),
            "must_include": all(s.lower() in (result.get("answer") or "").lower() for s in c.must_include),
            "must_not_include": not any(s.lower() in (result.get("answer") or "").lower() for s in c.must_not_include),
            "citation_valid": set(result.get("citations", [])).issubset(set(retrieved_ids)),
            "judge_score": judge(c, result) if c.expected_answer and not abstained else None,
            "cost_usd": result.get("cost_usd"), "latency_ms": result.get("latency_ms"),
        })
    return summarise(rows)                   # per query_type and overall; never just the mean

def summarise(rows):
    import pandas as pd
    df = pd.DataFrame(rows)
    overall = {"n": len(df), "recall@10": df["recall@10"].mean(),
               "behaviour_correct": df["behaviour_correct"].mean(),
               "must_include_pass": df["must_include"].mean(),
               "citation_valid": df["citation_valid"].mean(),
               "judge_mean": df["judge_score"].mean(),
               "p95_latency_ms": df["latency_ms"].quantile(0.95),
               "cost_per_query": df["cost_usd"].mean()}
    by_type = df.groupby("query_type").agg({"recall@10": "mean", "behaviour_correct": "mean",
                                            "judge_score": "mean"}).to_dict("index")
    return {"overall": overall, "by_query_type": by_type}       # breakdown is where the signal is

Pros

  • The only way to know whether a change helped. Every prompt, chunking, model, and retrieval decision in this reference is settled by running this.
  • Cheap relative to its value: a few person-days to build, then it pays forever.
  • Turns incidents into permanent protection — the same bug cannot ship twice.
  • Enables CI gating (the testing pyramid) so regressions are caught before users see them.
  • Per-category breakdowns localise problems (retrieval vs generation vs abstention).

Cons

  • Labelling is real work, and needs domain experts whose time is scarce.
  • Goes stale: as the corpus changes, ground-truth chunk IDs drift (a re-chunk invalidates them — store corpus_version and re-label or map).
  • Small sets have wide confidence intervals — a 3-point move on 50 cases means nothing (statistical hygiene).
  • Overfitting risk if you tune against the same set repeatedly without a held-out split.
  • Doesn't cover everything: rare tail cases and adversarial inputs need separate suites (the testing pyramid).

When to use

Before anything else. The first artifact of an AI system is not the retriever or the prompt; it's the golden set. Building it first also forces you to define what "correct" means, which is usually the hardest unresolved question in the project.

When NOT to use

  • Never skip it. But for a two-day prototype, 20 cases you wrote yourself is enough to start — just don't report percentages from it.

Production at scale

Stratified, not random. Sample by query type, source, tenant size, and language so that per-category metrics are meaningful. A set that's 90% factoid tells you nothing about multi-hop performance.

Corpus versioning is the maintenance burden. Ground-truth chunk IDs break when chunking config or parsers change (chunking, OCR / digitization). Two mitigations: label at document + span level and resolve to chunk IDs at eval time, or re-map IDs during migrations (versioning & migration). The span-level approach survives re-chunking and is worth the extra labelling effort.

Permission cases are mandatory in multi-tenant systems: the same question asked by two callers with different access, with different expected answers (authentication & authorization). These catch the leaks that no other test does.

Grow continuously, prune deliberately. Add from failures weekly (data flywheel); retire cases whose documents no longer exist. Target a steady size rather than unbounded growth — 200 well-chosen cases beat 2,000 stale ones.

Held-out split: 70% dev (used constantly), 30% test (run before releases only). Divergence between them is your overfitting signal.

Human agreement: double-label 10% and measure agreement. If experts disagree on the correct answer, your task definition is ambiguous and no model will satisfy it.

Cost and latency belong in the eval output, not just quality. A change that adds 4 points of accuracy and 3× the cost is a decision, not an automatic win (agent evals).

Make it one command. make eval producing a comparison against the previous run is the difference between a golden set that gets used and one that rots in a folder.

Prod checklist

  • 50–200+ real cases, stratified by query type, source, tenant, language
  • 15–20% unanswerable cases; multi-hop, aggregate, conflicting, and permission cases included
  • Labelled by domain experts; 10% double-labelled with agreement measured
  • Ground truth at document+span level, resolved to chunk IDs at eval time
  • Versioned dataset with corpus_version; results tied to a set version
  • Held-out test split not used during development
  • One-command run producing per-category breakdowns plus cost and latency
  • Weekly growth from production failures; stale cases retired
  • Wired into CI as a regression gate (the testing pyramid)

Metrics to watch

MetricWhy
Per-query-type scores (not just the mean)Where the system actually fails
Dev vs held-out test divergenceOverfitting
Inter-annotator agreementTask definition clarity
Golden-set size, age, and growth rateRepresentativeness
Cases traceable to production incidentsFlywheel health