Evaluation → Online eval (production signals)
What it is
Offline evals measure a frozen set of cases you chose. Production shows you the cases you didn't think of, the queries users actually ask, and quality under real retrieval, real load, and real corpus drift.
Signals, from weakest to strongest:
| Signal | Type | Strength | Notes |
|---|---|---|---|
| Thumbs up/down | Explicit | Weak — low volume (1–5% response rate), biased toward frustration | Useful for triage, not for measuring rates |
| Comment/report | Explicit | Strong per instance | Every one is a golden-set candidate |
| Copy / export | Implicit | Good positive signal | The user took the answer away |
| Edit distance between the answer and what the user shipped | Implicit | Strongest quality proxy in drafting products | Measures how wrong it was, not just that it was |
| Retry / rephrase | Implicit | Good negative signal | The answer didn't work |
| Abandon (no follow-up, session end) | Implicit | Weak-moderate | Ambiguous |
| Escalation to human | Implicit | Strong negative | Direct business metric |
| Task completion | Implicit | Strongest | Product-specific: ticket resolved, form submitted, deal drafted |
| Sampled LLM-judge scores | Automated | Moderate, calibrated (LLM-as-judge) | Continuous, covers everything |
| Automated checks | Automated | Strong for what they cover | Citation validity, faithfulness, schema — free on 100% of traffic |
The two that matter most in practice: automated checks on 100% of traffic (citation validity, faithfulness, abstention rate — all cheap) and implicit behavioural signals (edit distance, retry, escalation), because they're high-volume and unbiased by who bothers to click a thumb.
How to implement
- Instrument implicit signals first: copy, export, retry, rephrase, edit-then-send, escalate, session outcome. These need product instrumentation, not user cooperation.
- Run cheap automated checks online: citation ID validity, faithfulness (NLI), schema validity, abstention rate, retrieval top-score distribution — on every request, logged with the trace (observability & tracing).
- Sample for judge scoring: 1–5% of traffic scored against a rubric (LLM-as-judge), stratified by query type and tenant so rare categories are represented.
- Attach the version vector to every record: prompt, model, retrieval config, index version — so you can slice quality by version (prompt versioning & evaluation, versioning & rollout).
- Feed the flywheel: negative signals and judge-flagged cases go to a triage queue; confirmed failures become golden cases (data flywheel).
- Watch distributions, not just means: retrieval top-score, answer length, abstention rate, and cost per query all drift before quality visibly breaks.
- A/B by version for changes whose offline gain was marginal, with a pre-registered decision rule and adequate sample size (statistical hygiene).
Small example
from dataclasses import dataclass
@dataclass
class InteractionSignals:
trace_id: str; user_id: str; tenant_id: str
# version vector — slice everything by this
prompt_version: str; model_version: str; retrieval_config: str; index_version: str
# automated, computed on every request
citation_valid: bool; faithfulness: float; retrieval_top_score: float
abstained: bool; schema_valid: bool; cost_usd: float; latency_ms: float
# implicit, from product instrumentation
copied: bool = False; exported: bool = False; retried: bool = False
edited_before_use: bool = False; edit_distance: float | None = None
escalated_to_human: bool = False; task_completed: bool | None = None
# explicit
thumbs: int | None = None; comment: str | None = None
def online_checks(query, caller, retrieved, output) -> dict:
"""Cheap enough to run on 100% of traffic."""
valid_ids = {r["chunk_id"] for r in retrieved}
cited = output.get("citations", [])
claims = output.get("claims", [])
supported = sum(1 for cl in claims
if check_claim(cl["text"], "\n".join(r["text"] for r in retrieved
if r["chunk_id"] in cl["citations"]))["verdict"] == "entailed")
return {"citation_valid": set(cited).issubset(valid_ids),
"faithfulness": supported / len(claims) if claims else 1.0,
"retrieval_top_score": retrieved[0]["rerank_score"] if retrieved else 0.0,
"abstained": output.get("refused", False),
"schema_valid": output.get("schema_valid", True)}
def sample_for_judge(signals: InteractionSignals, base_rate=0.02) -> bool:
"""Stratified oversampling of the interesting cases."""
if signals.thumbs == -1 or signals.escalated_to_human or signals.retried: return True # always score negatives
if signals.faithfulness < 0.9 or not signals.citation_valid: return True # always score flagged
return random.random() < base_rate
def quality_by_version(records, metric="faithfulness", window="7d"):
import pandas as pd
df = pd.DataFrame([r.__dict__ for r in records])
return (df.groupby(["prompt_version", "model_version"])
.agg(n=("trace_id", "count"), mean=(metric, "mean"),
abstain=("abstained", "mean"), escalate=("escalated_to_human", "mean"),
cost=("cost_usd", "mean"))
.query("n >= 100")) # don't report on tiny slices (statistical hygiene)
def drift_alarm(records_today, records_baseline):
"""Distribution shifts precede visible quality drops."""
alarms = []
for metric, threshold in [("retrieval_top_score", -0.05), ("faithfulness", -0.03),
("abstained", 0.05), ("cost_usd", 0.20)]:
t = np.mean([getattr(r, metric) for r in records_today])
b = np.mean([getattr(r, metric) for r in records_baseline])
delta = (t - b) / b if metric == "cost_usd" else t - b
if (threshold < 0 and delta < threshold) or (threshold > 0 and delta > threshold):
alarms.append({"metric": metric, "baseline": round(b, 3), "today": round(t, 3), "delta": round(delta, 3)})
return alarms
Pros
- Measures reality, including the queries and corpus states your golden set never covered.
- Implicit signals are high-volume and unbiased compared with thumbs, which only frustrated users click.
- Automated checks on 100% of traffic catch regressions between releases, when offline evals aren't running.
- Distribution drift gives early warning — retrieval scores and abstention rates move before users complain.
- Feeds the flywheel: production failures become golden cases and training data (data flywheel).
Cons
- Implicit signals are ambiguous: a copy doesn't prove correctness; abandonment might mean satisfaction.
- No ground truth — you can measure faithfulness and behaviour, but not correctness, without human review.
- Explicit feedback is sparse and biased; rates computed from it are misleading.
- Privacy: logging queries and answers for evaluation is personal-data processing with retention and consent implications (data residency & retention, PII).
- Sampling cost for judge scoring, and the judge's own bias (LLM-as-judge).
- Attribution is hard: quality changes can come from the corpus, the model provider, or user behaviour, not just your deploys.
When to use
From launch. Automated online checks and implicit signals should ship with the first production version; judge sampling and A/B infrastructure can follow.
When NOT to use
- As a replacement for offline evals — online signals are noisy and lagging; you still need a gate before deploy (prompt evaluation).
- Explicit thumbs as a headline metric: report counts and use them for triage, not as a quality rate.
Production at scale
Automated checks on everything, judge on a sample. Citation validity, faithfulness, schema validity, and abstention are cheap enough for 100% coverage and catch the majority of regressions. Judge scoring is for the nuanced quality question, on 1–5%, stratified — and always oversampling negatives and flagged cases.
Edit distance is the best signal most products aren't capturing. In any drafting workflow (emails, summaries, code, reports), the difference between what you produced and what the user actually sent is the closest thing to a ground-truth quality measure that exists at scale. Instrument it.
Slice by version vector, always. "Faithfulness dropped 4 points" is unusable; "faithfulness dropped 4 points for prompt v7 on the Sonnet tier for tenant X" is actionable. This requires logging the full version vector on every interaction (observability & tracing).
Alert on distributions, not incidents. Retrieval top-score falling, abstention rate rising, answer length growing, cost per query climbing — each is an early indicator. Set baselines and alert on shifts (versioning & rollout).
Escalation-to-human and task-completion are the business metrics leadership funds (metrics cheat sheet). Tie them into the same dashboard as the technical metrics so quality work has a visible business narrative.
Privacy first: production evaluation stores prompts, answers, and retrieved content. Define retention, apply PII redaction where required, honour erasure, and keep tenant isolation in the eval store (data residency & retention, PII).
Continuous sampling into the golden set is the flywheel's main intake (data flywheel). Weekly: triage flagged cases, confirm the failure, write the expected answer, add it. This is what keeps offline evals representative.
Provider drift is real: an unpinned model changing behaviour shows up here first. Online monitoring is your detection mechanism for changes you didn't make (versioning & rollout).
Prod checklist
- Implicit signals instrumented: copy, export, retry, rephrase, edit-then-send with distance, escalate, task completion
- Automated checks on 100% of traffic: citation validity, faithfulness, schema, abstention, retrieval top score
- Stratified judge sampling (1–5%) oversampling negatives and flagged cases
- Full version vector logged with every interaction; all metrics sliceable by it
- Distribution-drift alarms with baselines
- Business metrics (escalation rate, task completion, deflection) on the same dashboard
- Retention, PII redaction, tenant isolation, and erasure for eval data
- Weekly triage of flagged cases into the golden set
- A/B capability with pre-registered decision rules and power analysis
Metrics to watch
| Metric | Why |
|---|---|
| Faithfulness and citation validity on live traffic | Continuous quality |
| Edit distance / retry / escalation rates | Real user-perceived quality |
| Abstention rate and retrieval top-score distribution | Corpus and retrieval drift |
| Task completion / deflection rate | The business number |
| Quality by version vector | Regression attribution |
Related
- Golden dataset — fed by online triage
- LLM-as-judge — sampled scoring
- Statistical hygiene — A/B sizing
- Observability — the trace and version vector
- Data flywheel, Triage
- Versioning & rollout — canary comparisons