Evaluation → Statistical hygiene
What it is
Eval numbers come from small samples of a stochastic system. Four disciplines separate signal from noise:
1. Control the randomness you can. Temperature 0 (or the lowest supported) for deterministic comparisons; pin model versions; cache retrieval so the retriever isn't a hidden variable; fix seeds where the API supports them. What remains is genuine model non-determinism, which is small at temperature 0 but not zero.
2. Measure the randomness you can't control. Where sampling matters (agents, temperature > 0, self-consistency), run N times and report the distribution, not one run.
3. Report confidence intervals. On 50 binary cases, the 95% CI on an 80% score is roughly ±11 points. A move from 80% to 83% is inside the noise. The Wilson interval is the right one for proportions at these sample sizes.
n=50, p=0.80 → 95% CI ≈ [0.67, 0.89] a 3-point move is nothing
n=200, p=0.80 → 95% CI ≈ [0.74, 0.85] a 3-point move is still marginal
n=1000,p=0.80 → 95% CI ≈ [0.77, 0.82] now a 3-point move is real
4. Use paired comparisons. Both variants run on the same cases, so you compare per-case outcomes rather than two independent means. McNemar's test on the disagreements is far more sensitive than comparing aggregate rates — and it's exactly the "which cases newly broke" question you already care about (prompt evaluation).
How to implement
- Freeze the eval configuration: temperature 0, pinned model version, pinned prompt version, cached retrieval results, fixed golden-set version.
- Paired design: same cases, both variants; record per-case pass/fail for each.
- Compute Wilson CIs for proportions and bootstrap CIs for judge-score means.
- McNemar's test on paired binary outcomes: only the discordant pairs (passed→failed, failed→passed) carry information.
- Report N, the CI, and the test result with every headline number. A metric without a sample size is not a metric.
- Multiple runs for stochastic components (agents, sampling): report mean, p95, and consistency across runs (agent evals).
- Correct for multiple comparisons when testing many variants or many metrics — otherwise one of them will look significant by chance.
Small example
import numpy as np
from scipy import stats
def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
"""Correct interval for proportions at small n — never use the normal approximation here."""
if n == 0: return (0.0, 0.0)
p = successes / n
denom = 1 + z**2 / n
centre = (p + z**2 / (2 * n)) / denom
half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
return (max(0.0, centre - half), min(1.0, centre + half))
def mcnemar(baseline: dict[str, bool], candidate: dict[str, bool]) -> dict:
"""Paired test on the SAME cases. Only disagreements carry information."""
b = sum(1 for k in baseline if baseline[k] and not candidate[k]) # regressions
c = sum(1 for k in baseline if not baseline[k] and candidate[k]) # fixes
if b + c == 0:
return {"regressions": 0, "fixes": 0, "p_value": 1.0, "verdict": "identical"}
# exact binomial test — appropriate for the small counts typical of eval sets
p = stats.binomtest(min(b, c), b + c, 0.5).pvalue
return {"regressions": b, "fixes": c, "p_value": round(p, 4),
"verdict": "significant" if p < 0.05 else "not distinguishable"}
def bootstrap_ci(scores: list[float], n_boot: int = 5000, alpha: float = 0.05) -> tuple[float, float]:
"""For continuous scores (judge ratings, faithfulness)."""
arr = np.array([s for s in scores if s is not None])
if len(arr) == 0: return (0.0, 0.0)
means = [np.mean(np.random.choice(arr, len(arr), replace=True)) for _ in range(n_boot)]
return (float(np.percentile(means, 100 * alpha / 2)), float(np.percentile(means, 100 * (1 - alpha / 2))))
def report(baseline_results, candidate_results):
b_pass = {r["case_id"]: r["passed"] for r in baseline_results}
c_pass = {r["case_id"]: r["passed"] for r in candidate_results}
n = len(c_pass)
b_rate, c_rate = sum(b_pass.values()) / n, sum(c_pass.values()) / n
test = mcnemar(b_pass, c_pass)
return {
"n": n,
"baseline": {"rate": round(b_rate, 3), "ci95": [round(x, 3) for x in wilson_ci(sum(b_pass.values()), n)]},
"candidate": {"rate": round(c_rate, 3), "ci95": [round(x, 3) for x in wilson_ci(sum(c_pass.values()), n)]},
"delta": round(c_rate - b_rate, 3),
"paired_test": test,
"judge_ci": [round(x, 3) for x in bootstrap_ci([r.get("judge_score") for r in candidate_results])],
"conclusion": ("SHIP" if test["verdict"] == "significant" and test["fixes"] > test["regressions"]
else "INCONCLUSIVE — need more cases or a larger effect"),
}
def required_n(baseline_rate=0.80, min_detectable_delta=0.05, power=0.8, alpha=0.05):
"""How many cases do you need to detect the improvement you care about?"""
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
es = proportion_effectsize(baseline_rate + min_detectable_delta, baseline_rate)
return int(np.ceil(NormalIndPower().solve_power(es, power=power, alpha=alpha, ratio=1)))
# baseline 0.80, detect +5 points → ~900 cases unpaired; far fewer paired
Pros
- Stops you shipping noise — the most common eval failure is celebrating a random fluctuation.
- Paired tests are far more sensitive than comparing means, so you can detect real improvements with modest sample sizes.
- CIs communicate honestly to stakeholders: "82% ± 5 on 200 cases" is a fact; "82%" is a claim.
- Power analysis tells you upfront whether your golden set can detect the improvement you're chasing.
- Costs nothing — it's arithmetic on results you already have.
Cons
- Small golden sets often can't reach significance, which is uncomfortable but true — and better known than not.
- Statistical rigour can slow decisions; sometimes shipping a plausible improvement with monitoring is the right call.
- Multiple-comparison corrections make many-variant sweeps look worse than they feel.
- Judge-based metrics have systematic bias that no confidence interval captures (LLM-as-judge) — CIs measure sampling noise, not bias.
When to use
Every reported eval comparison, especially before a release decision or a public claim. Also whenever someone says "this prompt is better" — ask for N and the interval.
When NOT to use (rigidly)
- Exploratory iteration: try things freely and eyeball results; apply rigour at the decision point.
- Obvious failures (schema violations, crashes, 0% pass) need no significance test.
- When the change is a safety fix — ship it and monitor; don't wait for statistical power on a security regression.
Production at scale
Report N and the interval on every dashboard number. A quality metric without a sample size invites false conclusions from everyone who reads it, including you in three months.
Paired comparison is the default form for prompt and model changes, and it's also the most operationally useful, since the list of regressions and fixes is the review artefact (prompt evaluation).
Size the golden set for the effect you care about. If you need to detect 3-point improvements, 50 cases will never do it. Run the power calculation and let it drive golden-set investment (golden dataset first).
Separate variance sources in stochastic systems: run-to-run variance (same input, multiple runs) and case-to-case variance. Agents need both reported (agent evals).
Temperature 0 is not fully deterministic on most providers (batching, hardware, routing). Expect small run-to-run differences even at temperature 0; if a "regression" is one case flipping, re-run before investigating.
Beware of the multiple-comparisons trap in sweeps: testing 20 prompt variants against a baseline at p<0.05 will produce one false winner on average. Apply a correction or treat sweeps as exploratory and confirm the winner in a fresh comparison.
Online A/B tests need their own power analysis (online eval): user-facing metrics are noisier and need larger samples than offline evals — usually thousands of sessions, not hundreds of cases.
Document the decision rule before running the test. "We ship if fixes exceed regressions with p<0.05, and no safety property regresses" prevents post-hoc rationalisation of a favourable-looking result.
Prod checklist
- Temperature 0, pinned model and prompt versions, cached retrieval for offline comparisons
- Paired design: both variants on the same cases; per-case outcomes recorded
- Wilson CIs for proportions; bootstrap CIs for continuous scores
- McNemar/exact binomial test on paired outcomes; regressions and fixes listed
- N, CI, and test result reported with every headline number
- Multiple runs with mean/p95/consistency for stochastic components
- Power analysis used to size the golden set for the target effect
- Multiple-comparison handling for variant sweeps
- Decision rule documented before the test is run
Metrics to watch
| Metric | Why |
|---|---|
| Confidence interval width on your headline metric | Whether your eval set can detect anything |
| Regressions vs fixes (paired) | The actual decision input |
| Run-to-run variance at temperature 0 | Baseline noise floor |
| Golden-set size vs minimum detectable effect | Investment guidance |
Related
- Golden dataset — sample size drives everything here
- Prompt evaluation — paired comparisons in practice
- LLM-as-judge — bias that CIs don't capture
- Agent evals — run-to-run variance
- Online eval — larger samples, noisier signals