Metrics cheat sheet
Video takeaway
"One metric per stage, each answering a specific question. Parsing accuracy caps everything downstream. Retrieval recall tells you whether the answer was even available. Reranking nDCG isolates the ranking gain. Faithfulness and citation validity cover hallucination. Then the one almost nobody has: correct-abstention and false-refusal rates, which need questions your corpus genuinely can't answer. For agents, always report cost and steps next to accuracy. And for leadership, report cost per successful task, not per request — a cheap system that fails half the time isn't cheap. Every number needs a sample size, an interval, and the ability to slice by which prompt and model produced it."
What it is
| Stage | Metric | Why it matters | Detail |
|---|---|---|---|
| Parsing | CER/WER, table-cell accuracy, reading-order accuracy | Caps everything downstream | Test-time validation against a golden sample |
| Chunking | Chunk-level recall on golden queries | Detects splits that orphan the answer | Chunking, component vs end-to-end |
| Retrieval | recall@k, precision@k, MRR, nDCG | Was the answer even present? | Retrieval evals |
| Reranking | nDCG@5 vs baseline, position of first relevant | Isolates the rerank gain | Reranking, retrieval evals |
| Generation | Faithfulness, answer relevance, citation accuracy | Hallucination and grounding | Generation evals |
| Refusal | Correct-abstention rate, false-refusal rate | The most-skipped RAG metric | Score thresholding & abstention, refusal / no-answer path |
| Agent | Task success, trajectory validity, steps/run, cost/run | Accuracy alone hides cost blowups | Agent evals |
| Production | p50/p95 latency, TTFT, $/request, error rate, cache-hit rate | SLO and unit economics | Cost levers, latency levers |
| Business | Deflection rate, time saved, human-escalation rate | The only numbers leadership funds | Online eval |
Three rules that make the table useful rather than decorative:
- Every metric has a sample size and a confidence interval (statistical hygiene). A number without N invites false conclusions.
- Every metric is sliceable by the version vector (observability & tracing) — prompt, model, index, tenant — or you cannot attribute a change.
- Report the distribution, not just the mean. p95 cost and p95 latency are where incidents live; means hide them.
How to implement
- Instrument once, derive many: the trace (observability & tracing) carries chunk IDs, scores, tokens, cost, latency, versions, and outcomes. Every metric here is a query over traces plus the golden set.
- Split offline and online: golden-set metrics gate releases (prompt evaluation, the testing pyramid); production metrics detect drift and incidents (online eval).
- One dashboard per audience: engineering (per-stage quality, latency, cost), operations (SLOs, error and degradation rates), leadership (business outcomes and $/task).
- Alert on distributions, not just failures: retrieval top score, cache hit rate, abstention rate, and cost per request all shift before users complain (versioning & rollout).
- Set targets per metric and record the rationale; a metric with no target generates no decisions.
- Review weekly alongside the failure-bucket histogram (feedback capture & triage).
Small example
METRIC_SPEC = {
# stage: (metric, source, target, alert_rule)
"parsing": ("table_cell_accuracy", "golden_parse", 0.90, "drop > 0.02"),
"chunking": ("chunk_recall@10", "golden_retr", 0.90, "drop > 0.03"),
"retrieval": ("recall@50", "golden_retr", 0.90, "drop > 0.03"),
"reranking": ("ndcg@6", "golden_retr", 0.75, "drop > 0.03"),
"generation": ("faithfulness", "prod_sampled", 0.95, "drop > 0.03"),
"citation": ("citation_valid_rate", "prod_all", 0.99, "drop > 0.01"),
"refusal_correct":("correct_abstention_rate", "golden_unans", 0.85, "drop > 0.05"),
"refusal_false": ("false_refusal_rate", "golden_ans", 0.05, "rise > 0.03"),
"agent": ("task_success_rate", "agent_golden", 0.80, "drop > 0.05"),
"agent_cost": ("cost_per_run_p95", "prod_all", 0.50, "rise > 30%"),
"latency": ("ttft_p95_ms", "prod_all", 800, "rise > 30%"),
"cost": ("cost_per_request_p95", "prod_all", 0.05, "rise > 30%"),
"cache": ("cache_hit_rate", "prod_all", 0.70, "drop > 0.15"),
"business": ("deflection_rate", "prod_all", 0.60, "drop > 0.05"),
}
def wilson_ci(successes, n, z=1.96):
if n == 0: return (0.0, 0.0)
p = successes / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d
h = z * ((p*(1-p)/n + z*z/(4*n*n)) ** 0.5) / d
return (max(0, c-h), min(1, c+h))
def metric_report(window="7d", slice_by=("prompt_version", "model")) -> dict:
"""Every number carries N and an interval, and is sliceable by version (statistical hygiene, observability & tracing)."""
rows = []
for stage, (name, source, target, alert) in METRIC_SPEC.items():
vals = query_metric(name, source, window)
n, mean = len(vals), (sum(vals) / len(vals) if vals else None)
ci = wilson_ci(sum(1 for v in vals if v >= 0.5), n) if is_binary(name) else bootstrap_ci(vals)
rows.append({"stage": stage, "metric": name, "n": n,
"value": round(mean, 4) if mean is not None else None,
"ci95": [round(x, 4) for x in ci], "target": target,
"meets_target": (mean is not None and meets(mean, target, name)),
"by_slice": {s: group_metric(name, source, window, s) for s in slice_by}})
return {"window": window, "rows": rows,
"failing": [r["metric"] for r in rows if not r["meets_target"]]}
def drift_alerts(current: dict, baseline: dict) -> list[dict]:
out = []
for stage, (name, _, target, rule) in METRIC_SPEC.items():
c, b = current.get(name), baseline.get(name)
if c is None or b is None: continue
if rule.startswith("drop") and b - c > float(rule.split(">")[1]):
out.append({"metric": name, "baseline": b, "current": c, "direction": "down"})
if rule.startswith("rise") and "%" in rule and c > b * (1 + float(rule.split(">")[1].strip(" %"))/100):
out.append({"metric": name, "baseline": b, "current": c, "direction": "up"})
return out
# The three dashboards, from one source
ENGINEERING = ["chunk_recall@10", "recall@50", "ndcg@6", "faithfulness", "citation_valid_rate"]
OPERATIONS = ["ttft_p95_ms", "cost_per_request_p95", "error_rate", "degraded_rate", "cache_hit_rate"]
LEADERSHIP = ["deflection_rate", "human_escalation_rate", "cost_per_successful_task", "time_saved_hours"]
Pros
- One metric per stage makes attribution possible — a drop localises to a stage rather than to "the system".
- Deterministic metrics (parsing, retrieval, citations) are cheap and can run continuously.
- Targets turn measurements into decisions and into release gates (the testing pyramid).
- Three dashboards for three audiences keeps each conversation grounded in the right numbers.
- Business metrics connect engineering work to funding.
Cons
- Metric sprawl: too many numbers and nobody watches any of them. Keep one primary per stage.
- Goodhart's law: optimising a proxy (faithfulness) can degrade the real goal (usefulness) — pair quality metrics with business outcomes.
- Small samples give wide intervals and invite over-reaction (statistical hygiene).
- Judge-based metrics carry systematic bias that intervals don't capture (LLM-as-judge).
- Business metrics are lagging and confounded by product changes outside the AI system.
When to use
From the first production deployment. Start with the deterministic ones — citation validity, retrieval recall, latency, cost — which are free and catch most regressions.
When NOT to use (all of them)
- A prototype needs golden-set quality plus cost and latency; the full table is for production.
- Don't add agent metrics if you have no agents, or graph metrics with no graph.
Production at scale
Refusal metrics are the ones nobody has (metrics cheat sheet says so explicitly). Correct-abstention and false-refusal rates require unanswerable questions in the golden set (golden dataset first), which most teams never build. They are also among the most predictive of user trust.
Cost per successful task, not per request (cost levers). A cheap system that fails half the time is expensive, and this is the number to put in front of leadership alongside deflection.
Slice everything by version vector. "Faithfulness dropped three points" is unactionable; "faithfulness dropped three points on prompt v8 for the Sonnet tier" is a rollback decision (versioning & rollout).
Alert on leading indicators: retrieval top-score distribution, cache hit rate, abstention rate, and p95 cost all move before user-visible quality does.
Distinguish gate metrics from monitoring metrics. Gate metrics (golden-set quality, safety) block releases (the testing pyramid). Monitoring metrics (production distributions) trigger investigation. Confusing them either blocks releases on noise or lets regressions through.
Keep the primary metric per stage stable over time — changing definitions destroys trend data, which is often more valuable than the absolute value.
Pair every quality metric with a cost metric in the same view, so trade-offs are visible at the moment of decision (agent evals).
Review weekly with the failure-bucket histogram (feedback capture & triage): metrics say what moved, buckets say where to fix it.
Prod checklist
- One primary metric per stage, with a target and a documented rationale
- Every metric reported with N and a confidence interval
- All metrics sliceable by version vector and tenant
- Gate metrics (release-blocking) separated from monitoring metrics
- Refusal metrics implemented, requiring unanswerable golden cases
- Cost per successful task computed and reported
- Three dashboards: engineering, operations, leadership
- Alerts on leading indicators and distribution shifts, not just failures
- Metric definitions versioned and stable over time
- Weekly review alongside the failure-bucket histogram
Metrics to watch (the meta-metrics)
| Metric | Why |
|---|---|
| Metrics meeting target / total | System health at a glance |
| Metrics with N below usable threshold | Where you can't conclude anything |
| Alert precision (alerts that led to action) | Whether the alerting is trusted |
| Time from metric drop to diagnosis | Observability effectiveness |
Related
- Evaluation — how each metric is computed
- Observability — the data source
- Cost levers / latency levers — cost and latency metrics
- Triage — the histogram that complements this table
- Statistical hygiene — how to read the numbers