Evaluation → Online eval (production signals)

Online Evaluation for LLM Apps Explained: Production Signals

What it is

Offline evals measure a frozen set of cases you chose. Production shows you the cases you didn't think of, the queries users actually ask, and quality under real retrieval, real load, and real corpus drift.

Signals, from weakest to strongest:

SignalTypeStrengthNotes
Thumbs up/downExplicitWeak — low volume (1–5% response rate), biased toward frustrationUseful for triage, not for measuring rates
Comment/reportExplicitStrong per instanceEvery one is a golden-set candidate
Copy / exportImplicitGood positive signalThe user took the answer away
Edit distance between the answer and what the user shippedImplicitStrongest quality proxy in drafting productsMeasures how wrong it was, not just that it was
Retry / rephraseImplicitGood negative signalThe answer didn't work
Abandon (no follow-up, session end)ImplicitWeak-moderateAmbiguous
Escalation to humanImplicitStrong negativeDirect business metric
Task completionImplicitStrongestProduct-specific: ticket resolved, form submitted, deal drafted
Sampled LLM-judge scoresAutomatedModerate, calibrated (LLM-as-judge)Continuous, covers everything
Automated checksAutomatedStrong for what they coverCitation validity, faithfulness, schema — free on 100% of traffic

The two that matter most in practice: automated checks on 100% of traffic (citation validity, faithfulness, abstention rate — all cheap) and implicit behavioural signals (edit distance, retry, escalation), because they're high-volume and unbiased by who bothers to click a thumb.

How to implement

  1. Instrument implicit signals first: copy, export, retry, rephrase, edit-then-send, escalate, session outcome. These need product instrumentation, not user cooperation.
  2. Run cheap automated checks online: citation ID validity, faithfulness (NLI), schema validity, abstention rate, retrieval top-score distribution — on every request, logged with the trace (observability & tracing).
  3. Sample for judge scoring: 1–5% of traffic scored against a rubric (LLM-as-judge), stratified by query type and tenant so rare categories are represented.
  4. Attach the version vector to every record: prompt, model, retrieval config, index version — so you can slice quality by version (prompt versioning & evaluation, versioning & rollout).
  5. Feed the flywheel: negative signals and judge-flagged cases go to a triage queue; confirmed failures become golden cases (data flywheel).
  6. Watch distributions, not just means: retrieval top-score, answer length, abstention rate, and cost per query all drift before quality visibly breaks.
  7. A/B by version for changes whose offline gain was marginal, with a pre-registered decision rule and adequate sample size (statistical hygiene).

Small example

from dataclasses import dataclass

@dataclass
class InteractionSignals:
    trace_id: str; user_id: str; tenant_id: str
    # version vector — slice everything by this
    prompt_version: str; model_version: str; retrieval_config: str; index_version: str
    # automated, computed on every request
    citation_valid: bool; faithfulness: float; retrieval_top_score: float
    abstained: bool; schema_valid: bool; cost_usd: float; latency_ms: float
    # implicit, from product instrumentation
    copied: bool = False; exported: bool = False; retried: bool = False
    edited_before_use: bool = False; edit_distance: float | None = None
    escalated_to_human: bool = False; task_completed: bool | None = None
    # explicit
    thumbs: int | None = None; comment: str | None = None

def online_checks(query, caller, retrieved, output) -> dict:
    """Cheap enough to run on 100% of traffic."""
    valid_ids = {r["chunk_id"] for r in retrieved}
    cited = output.get("citations", [])
    claims = output.get("claims", [])
    supported = sum(1 for cl in claims
                    if check_claim(cl["text"], "\n".join(r["text"] for r in retrieved
                                                         if r["chunk_id"] in cl["citations"]))["verdict"] == "entailed")
    return {"citation_valid": set(cited).issubset(valid_ids),
            "faithfulness": supported / len(claims) if claims else 1.0,
            "retrieval_top_score": retrieved[0]["rerank_score"] if retrieved else 0.0,
            "abstained": output.get("refused", False),
            "schema_valid": output.get("schema_valid", True)}

def sample_for_judge(signals: InteractionSignals, base_rate=0.02) -> bool:
    """Stratified oversampling of the interesting cases."""
    if signals.thumbs == -1 or signals.escalated_to_human or signals.retried:  return True   # always score negatives
    if signals.faithfulness < 0.9 or not signals.citation_valid:               return True   # always score flagged
    return random.random() < base_rate

def quality_by_version(records, metric="faithfulness", window="7d"):
    import pandas as pd
    df = pd.DataFrame([r.__dict__ for r in records])
    return (df.groupby(["prompt_version", "model_version"])
              .agg(n=("trace_id", "count"), mean=(metric, "mean"),
                   abstain=("abstained", "mean"), escalate=("escalated_to_human", "mean"),
                   cost=("cost_usd", "mean"))
              .query("n >= 100"))                                    # don't report on tiny slices (statistical hygiene)

def drift_alarm(records_today, records_baseline):
    """Distribution shifts precede visible quality drops."""
    alarms = []
    for metric, threshold in [("retrieval_top_score", -0.05), ("faithfulness", -0.03),
                              ("abstained", 0.05), ("cost_usd", 0.20)]:
        t = np.mean([getattr(r, metric) for r in records_today])
        b = np.mean([getattr(r, metric) for r in records_baseline])
        delta = (t - b) / b if metric == "cost_usd" else t - b
        if (threshold < 0 and delta < threshold) or (threshold > 0 and delta > threshold):
            alarms.append({"metric": metric, "baseline": round(b, 3), "today": round(t, 3), "delta": round(delta, 3)})
    return alarms

Pros

  • Measures reality, including the queries and corpus states your golden set never covered.
  • Implicit signals are high-volume and unbiased compared with thumbs, which only frustrated users click.
  • Automated checks on 100% of traffic catch regressions between releases, when offline evals aren't running.
  • Distribution drift gives early warning — retrieval scores and abstention rates move before users complain.
  • Feeds the flywheel: production failures become golden cases and training data (data flywheel).

Cons

  • Implicit signals are ambiguous: a copy doesn't prove correctness; abandonment might mean satisfaction.
  • No ground truth — you can measure faithfulness and behaviour, but not correctness, without human review.
  • Explicit feedback is sparse and biased; rates computed from it are misleading.
  • Privacy: logging queries and answers for evaluation is personal-data processing with retention and consent implications (data residency & retention, PII).
  • Sampling cost for judge scoring, and the judge's own bias (LLM-as-judge).
  • Attribution is hard: quality changes can come from the corpus, the model provider, or user behaviour, not just your deploys.

When to use

From launch. Automated online checks and implicit signals should ship with the first production version; judge sampling and A/B infrastructure can follow.

When NOT to use

  • As a replacement for offline evals — online signals are noisy and lagging; you still need a gate before deploy (prompt evaluation).
  • Explicit thumbs as a headline metric: report counts and use them for triage, not as a quality rate.

Production at scale

Automated checks on everything, judge on a sample. Citation validity, faithfulness, schema validity, and abstention are cheap enough for 100% coverage and catch the majority of regressions. Judge scoring is for the nuanced quality question, on 1–5%, stratified — and always oversampling negatives and flagged cases.

Edit distance is the best signal most products aren't capturing. In any drafting workflow (emails, summaries, code, reports), the difference between what you produced and what the user actually sent is the closest thing to a ground-truth quality measure that exists at scale. Instrument it.

Slice by version vector, always. "Faithfulness dropped 4 points" is unusable; "faithfulness dropped 4 points for prompt v7 on the Sonnet tier for tenant X" is actionable. This requires logging the full version vector on every interaction (observability & tracing).

Alert on distributions, not incidents. Retrieval top-score falling, abstention rate rising, answer length growing, cost per query climbing — each is an early indicator. Set baselines and alert on shifts (versioning & rollout).

Escalation-to-human and task-completion are the business metrics leadership funds (metrics cheat sheet). Tie them into the same dashboard as the technical metrics so quality work has a visible business narrative.

Privacy first: production evaluation stores prompts, answers, and retrieved content. Define retention, apply PII redaction where required, honour erasure, and keep tenant isolation in the eval store (data residency & retention, PII).

Continuous sampling into the golden set is the flywheel's main intake (data flywheel). Weekly: triage flagged cases, confirm the failure, write the expected answer, add it. This is what keeps offline evals representative.

Provider drift is real: an unpinned model changing behaviour shows up here first. Online monitoring is your detection mechanism for changes you didn't make (versioning & rollout).

Prod checklist

  • Implicit signals instrumented: copy, export, retry, rephrase, edit-then-send with distance, escalate, task completion
  • Automated checks on 100% of traffic: citation validity, faithfulness, schema, abstention, retrieval top score
  • Stratified judge sampling (1–5%) oversampling negatives and flagged cases
  • Full version vector logged with every interaction; all metrics sliceable by it
  • Distribution-drift alarms with baselines
  • Business metrics (escalation rate, task completion, deflection) on the same dashboard
  • Retention, PII redaction, tenant isolation, and erasure for eval data
  • Weekly triage of flagged cases into the golden set
  • A/B capability with pre-registered decision rules and power analysis

Metrics to watch

MetricWhy
Faithfulness and citation validity on live trafficContinuous quality
Edit distance / retry / escalation ratesReal user-perceived quality
Abstention rate and retrieval top-score distributionCorpus and retrieval drift
Task completion / deflection rateThe business number
Quality by version vectorRegression attribution