Metrics cheat sheet

Video takeaway

"One metric per stage, each answering a specific question. Parsing accuracy caps everything downstream. Retrieval recall tells you whether the answer was even available. Reranking nDCG isolates the ranking gain. Faithfulness and citation validity cover hallucination. Then the one almost nobody has: correct-abstention and false-refusal rates, which need questions your corpus genuinely can't answer. For agents, always report cost and steps next to accuracy. And for leadership, report cost per successful task, not per request — a cheap system that fails half the time isn't cheap. Every number needs a sample size, an interval, and the ability to slice by which prompt and model produced it."

What it is

StageMetricWhy it mattersDetail
ParsingCER/WER, table-cell accuracy, reading-order accuracyCaps everything downstreamTest-time validation against a golden sample
ChunkingChunk-level recall on golden queriesDetects splits that orphan the answerChunking, component vs end-to-end
Retrievalrecall@k, precision@k, MRR, nDCGWas the answer even present?Retrieval evals
RerankingnDCG@5 vs baseline, position of first relevantIsolates the rerank gainReranking, retrieval evals
GenerationFaithfulness, answer relevance, citation accuracyHallucination and groundingGeneration evals
RefusalCorrect-abstention rate, false-refusal rateThe most-skipped RAG metricScore thresholding & abstention, refusal / no-answer path
AgentTask success, trajectory validity, steps/run, cost/runAccuracy alone hides cost blowupsAgent evals
Productionp50/p95 latency, TTFT, $/request, error rate, cache-hit rateSLO and unit economicsCost levers, latency levers
BusinessDeflection rate, time saved, human-escalation rateThe only numbers leadership fundsOnline eval

Three rules that make the table useful rather than decorative:

  1. Every metric has a sample size and a confidence interval (statistical hygiene). A number without N invites false conclusions.
  2. Every metric is sliceable by the version vector (observability & tracing) — prompt, model, index, tenant — or you cannot attribute a change.
  3. Report the distribution, not just the mean. p95 cost and p95 latency are where incidents live; means hide them.

How to implement

  1. Instrument once, derive many: the trace (observability & tracing) carries chunk IDs, scores, tokens, cost, latency, versions, and outcomes. Every metric here is a query over traces plus the golden set.
  2. Split offline and online: golden-set metrics gate releases (prompt evaluation, the testing pyramid); production metrics detect drift and incidents (online eval).
  3. One dashboard per audience: engineering (per-stage quality, latency, cost), operations (SLOs, error and degradation rates), leadership (business outcomes and $/task).
  4. Alert on distributions, not just failures: retrieval top score, cache hit rate, abstention rate, and cost per request all shift before users complain (versioning & rollout).
  5. Set targets per metric and record the rationale; a metric with no target generates no decisions.
  6. Review weekly alongside the failure-bucket histogram (feedback capture & triage).

Small example

METRIC_SPEC = {
    # stage:          (metric,                     source,          target,  alert_rule)
    "parsing":        ("table_cell_accuracy",      "golden_parse",  0.90,   "drop > 0.02"),
    "chunking":       ("chunk_recall@10",          "golden_retr",   0.90,   "drop > 0.03"),
    "retrieval":      ("recall@50",                "golden_retr",   0.90,   "drop > 0.03"),
    "reranking":      ("ndcg@6",                   "golden_retr",   0.75,   "drop > 0.03"),
    "generation":     ("faithfulness",             "prod_sampled",  0.95,   "drop > 0.03"),
    "citation":       ("citation_valid_rate",      "prod_all",      0.99,   "drop > 0.01"),
    "refusal_correct":("correct_abstention_rate",  "golden_unans",  0.85,   "drop > 0.05"),
    "refusal_false":  ("false_refusal_rate",       "golden_ans",    0.05,   "rise > 0.03"),
    "agent":          ("task_success_rate",        "agent_golden",  0.80,   "drop > 0.05"),
    "agent_cost":     ("cost_per_run_p95",         "prod_all",      0.50,   "rise > 30%"),
    "latency":        ("ttft_p95_ms",              "prod_all",      800,    "rise > 30%"),
    "cost":           ("cost_per_request_p95",     "prod_all",      0.05,   "rise > 30%"),
    "cache":          ("cache_hit_rate",           "prod_all",      0.70,   "drop > 0.15"),
    "business":       ("deflection_rate",          "prod_all",      0.60,   "drop > 0.05"),
}

def wilson_ci(successes, n, z=1.96):
    if n == 0: return (0.0, 0.0)
    p = successes / n; d = 1 + z*z/n
    c = (p + z*z/(2*n)) / d
    h = z * ((p*(1-p)/n + z*z/(4*n*n)) ** 0.5) / d
    return (max(0, c-h), min(1, c+h))

def metric_report(window="7d", slice_by=("prompt_version", "model")) -> dict:
    """Every number carries N and an interval, and is sliceable by version (statistical hygiene, observability & tracing)."""
    rows = []
    for stage, (name, source, target, alert) in METRIC_SPEC.items():
        vals = query_metric(name, source, window)
        n, mean = len(vals), (sum(vals) / len(vals) if vals else None)
        ci = wilson_ci(sum(1 for v in vals if v >= 0.5), n) if is_binary(name) else bootstrap_ci(vals)
        rows.append({"stage": stage, "metric": name, "n": n,
                     "value": round(mean, 4) if mean is not None else None,
                     "ci95": [round(x, 4) for x in ci], "target": target,
                     "meets_target": (mean is not None and meets(mean, target, name)),
                     "by_slice": {s: group_metric(name, source, window, s) for s in slice_by}})
    return {"window": window, "rows": rows,
            "failing": [r["metric"] for r in rows if not r["meets_target"]]}

def drift_alerts(current: dict, baseline: dict) -> list[dict]:
    out = []
    for stage, (name, _, target, rule) in METRIC_SPEC.items():
        c, b = current.get(name), baseline.get(name)
        if c is None or b is None: continue
        if rule.startswith("drop") and b - c > float(rule.split(">")[1]):
            out.append({"metric": name, "baseline": b, "current": c, "direction": "down"})
        if rule.startswith("rise") and "%" in rule and c > b * (1 + float(rule.split(">")[1].strip(" %"))/100):
            out.append({"metric": name, "baseline": b, "current": c, "direction": "up"})
    return out

# The three dashboards, from one source
ENGINEERING = ["chunk_recall@10", "recall@50", "ndcg@6", "faithfulness", "citation_valid_rate"]
OPERATIONS  = ["ttft_p95_ms", "cost_per_request_p95", "error_rate", "degraded_rate", "cache_hit_rate"]
LEADERSHIP  = ["deflection_rate", "human_escalation_rate", "cost_per_successful_task", "time_saved_hours"]

Pros

  • One metric per stage makes attribution possible — a drop localises to a stage rather than to "the system".
  • Deterministic metrics (parsing, retrieval, citations) are cheap and can run continuously.
  • Targets turn measurements into decisions and into release gates (the testing pyramid).
  • Three dashboards for three audiences keeps each conversation grounded in the right numbers.
  • Business metrics connect engineering work to funding.

Cons

  • Metric sprawl: too many numbers and nobody watches any of them. Keep one primary per stage.
  • Goodhart's law: optimising a proxy (faithfulness) can degrade the real goal (usefulness) — pair quality metrics with business outcomes.
  • Small samples give wide intervals and invite over-reaction (statistical hygiene).
  • Judge-based metrics carry systematic bias that intervals don't capture (LLM-as-judge).
  • Business metrics are lagging and confounded by product changes outside the AI system.

When to use

From the first production deployment. Start with the deterministic ones — citation validity, retrieval recall, latency, cost — which are free and catch most regressions.

When NOT to use (all of them)

  • A prototype needs golden-set quality plus cost and latency; the full table is for production.
  • Don't add agent metrics if you have no agents, or graph metrics with no graph.

Production at scale

Refusal metrics are the ones nobody has (metrics cheat sheet says so explicitly). Correct-abstention and false-refusal rates require unanswerable questions in the golden set (golden dataset first), which most teams never build. They are also among the most predictive of user trust.

Cost per successful task, not per request (cost levers). A cheap system that fails half the time is expensive, and this is the number to put in front of leadership alongside deflection.

Slice everything by version vector. "Faithfulness dropped three points" is unactionable; "faithfulness dropped three points on prompt v8 for the Sonnet tier" is a rollback decision (versioning & rollout).

Alert on leading indicators: retrieval top-score distribution, cache hit rate, abstention rate, and p95 cost all move before user-visible quality does.

Distinguish gate metrics from monitoring metrics. Gate metrics (golden-set quality, safety) block releases (the testing pyramid). Monitoring metrics (production distributions) trigger investigation. Confusing them either blocks releases on noise or lets regressions through.

Keep the primary metric per stage stable over time — changing definitions destroys trend data, which is often more valuable than the absolute value.

Pair every quality metric with a cost metric in the same view, so trade-offs are visible at the moment of decision (agent evals).

Review weekly with the failure-bucket histogram (feedback capture & triage): metrics say what moved, buckets say where to fix it.

Prod checklist

  • One primary metric per stage, with a target and a documented rationale
  • Every metric reported with N and a confidence interval
  • All metrics sliceable by version vector and tenant
  • Gate metrics (release-blocking) separated from monitoring metrics
  • Refusal metrics implemented, requiring unanswerable golden cases
  • Cost per successful task computed and reported
  • Three dashboards: engineering, operations, leadership
  • Alerts on leading indicators and distribution shifts, not just failures
  • Metric definitions versioned and stable over time
  • Weekly review alongside the failure-bucket histogram

Metrics to watch (the meta-metrics)

MetricWhy
Metrics meeting target / totalSystem health at a glance
Metrics with N below usable thresholdWhere you can't conclude anything
Alert precision (alerts that led to action)Whether the alerting is trusted
Time from metric drop to diagnosisObservability effectiveness