Evaluation → LLM-as-judge
What it is
Open-ended outputs — summaries, explanations, drafted replies — have no exact match. Human grading doesn't scale. LLM-as-judge fills the gap: a model scores outputs against explicit criteria, cheaply, on thousands of cases.
Two modes:
| Mode | How | Strengths | Weaknesses |
|---|---|---|---|
| Rubric / pointwise | Score one output on defined criteria (1–5, or pass/fail per criterion) | Absolute scores, comparable across runs, interpretable | Scores drift; models cluster around 3–4; poor discrimination |
| Pairwise | Show two outputs, ask which is better | Much more reliable; humans and models both judge comparisons better than absolutes | No absolute score; O(n²) comparisons; position bias |
The known biases are well documented and must be controlled:
- Position bias: the first (or last) option is favoured — randomise order and evaluate both orderings.
- Length bias: longer answers score higher regardless of quality — control by including conciseness in the rubric and checking score-vs-length correlation.
- Self-preference: models prefer their own outputs — use a different model family as judge than the one generating.
- Sycophancy / style over substance: confident, well-formatted, wrong answers outscore hedged, correct ones — anchor the rubric to the ground-truth answer.
The non-negotiable step is calibration: measure agreement between the judge and human labels on a sample, and report it alongside every judge-derived metric.
How to implement
- Write a rubric with concrete criteria, each independently checkable, anchored to the golden case's expected answer — not "is this good", but "does it state the notice period correctly", "does it cite a passage that supports it", "does it avoid claims not in the context".
- Use a different model family from the generator; a strong model, temperature 0.
- Provide the reference answer where you have one (golden dataset first) — reference-based judging is far more reliable than reference-free.
- Force structured output with per-criterion verdicts and a short reason (structured outputs & tool use), so scores are auditable and debuggable.
- Control position bias in pairwise: run both orders, count a win only if consistent; ties otherwise.
- Calibrate: 50–100 cases labelled by humans; compute agreement (accuracy, Cohen's κ, or Spearman correlation for scores). Report it with every result.
- Recalibrate on judge-model changes, rubric changes, and periodically.
Small example
RUBRIC_SYS = """You grade an assistant's answer against a reference answer and the retrieved context.
Grade each criterion independently. Be strict: a plausible answer that is not supported by the context fails C3.
Criteria:
C1 correctness — states the same facts as the reference (numbers, dates, entities must match exactly)
C2 completeness — addresses every part of the question
C3 groundedness — every factual claim is supported by the provided context
C4 citation — cited passage ids exist and support the claims attributed to them
C5 conciseness — no padding, hedging, or restating the question
Output JSON:
{"C1": {"pass": bool, "reason": string}, ..., "C5": {...},
"overall": 1-5, "critical_failure": bool}
Set critical_failure=true if C1 or C3 fails."""
def judge_rubric(query, reference, answer, context, judge_model="claude-opus-5"):
out = json.loads(client.messages.create(
model=judge_model, max_tokens=700, temperature=0, system=RUBRIC_SYS,
messages=[{"role": "user", "content":
f"<question>{query}</question>\n<reference_answer>{reference}</reference_answer>\n"
f"<context>{context}</context>\n<answer>{answer}</answer>"}]).content[0].text)
return out
def judge_pairwise(query, a, b, reference, judge_model="claude-opus-5"):
"""Position-bias control: judge both orders; a win requires consistency."""
def one(first, second):
r = client.messages.create(model=judge_model, max_tokens=300, temperature=0, system=PAIRWISE_SYS,
messages=[{"role": "user", "content":
f"<question>{query}</question><reference>{reference}</reference>"
f"<answer_1>{first}</answer_1><answer_2>{second}</answer_2>"}]).content[0].text
return json.loads(r)["winner"] # "1" | "2" | "tie"
ab, ba = one(a, b), one(b, a)
if ab == "1" and ba == "2": return "A"
if ab == "2" and ba == "1": return "B"
return "tie" # inconsistent → tie, and log it
# --- calibration: the step that makes judge scores usable ---
def calibrate(judge_fn, human_labelled: list[dict]) -> dict:
from sklearn.metrics import cohen_kappa_score
j = [judge_fn(x["query"], x["reference"], x["answer"], x["context"])["overall"] >= 4 for x in human_labelled]
h = [x["human_pass"] for x in human_labelled]
agreement = sum(a == b for a, b in zip(j, h)) / len(h)
kappa = cohen_kappa_score(h, j)
fp = sum(1 for a, b in zip(j, h) if a and not b) / max(sum(j), 1) # judge passed, human failed
fn = sum(1 for a, b in zip(j, h) if not a and b) / max(len(h) - sum(j), 1)
return {"agreement": round(agreement, 3), "kappa": round(kappa, 3),
"false_pass_rate": round(fp, 3), "false_fail_rate": round(fn, 3), "n": len(h)}
def check_length_bias(results):
import numpy as np
lens = [len(r["answer"].split()) for r in results]
scores = [r["judge"]["overall"] for r in results]
return float(np.corrcoef(lens, scores)[0, 1]) # |r| > 0.3 → investigate the rubric
Pros
- Scales grading to thousands of cases at a fraction of human cost.
- Consistent application of a rubric (no rater fatigue or drift within a run).
- Per-criterion output is diagnostic — you learn why an answer failed, not just that it did.
- Pairwise mode is reliable enough for A/B decisions between prompt or model versions.
- Fast enough to run in CI (prompt evaluation).
Cons
- Systematic biases (position, length, self-preference, style) that correlate across all cases — they don't average out.
- Uncalibrated judge scores are meaningless numbers that teams nonetheless put in dashboards and reports.
- Cost: a strong judge on every case doubles or triples eval cost.
- Judges miss domain errors experts would catch — a plausible-sounding legal or medical error can pass.
- Drift: the judge model changes under you (pin versions).
- Cannot be the acceptance criterion for anything high-stakes.
When to use
- Open-ended generation quality where deterministic checks can't decide.
- Comparing prompt or model variants at scale (pairwise is best here).
- Continuous monitoring of production samples (online eval).
- Grading the outputs of other components (critic quality, loop-back / reflection; grader quality, corrective RAG (CRAG): grade, then rewrite or fall back).
When NOT to use
- As sole truth for high-stakes decisions — regulatory, medical, legal, or financial correctness needs human sign-off.
- When a deterministic check exists — never pay a judge to verify a citation ID exists.
- With the same model as generator (self-preference), or without calibration data.
- To compare across time when the judge model has changed — re-baseline first.
Production at scale
Report agreement alongside every judge metric. "Judge score 4.2 (agreement with human labels: 0.83, n=80)" is a usable number. "Judge score 4.2" is not. Publish the false-pass and false-fail rates too — false-pass is the dangerous direction, because it hides real failures.
Pin the judge model and rubric version. A judge upgrade shifts all scores; historical comparisons break silently. Treat judge changes like model changes: re-baseline, note the discontinuity (versioning & rollout).
Different family for judging. Using the generator's own model family inflates its scores. If you serve Claude, judge with a different provider's model, or at minimum a different size class, and measure self-preference directly by having each judge grade both systems' outputs.
Prefer pairwise for decisions, rubric for tracking. Pairwise is more reliable for "is B better than A"; rubric scores are better for a dashboard trend line. Many teams run both.
Anchor to references. Reference-based judging (with the golden answer supplied) dramatically outperforms reference-free. This is another reason the golden set (golden dataset first) is the foundation.
Sample, don't grade everything. For production monitoring, a 1–5% sample with confidence intervals costs a fraction and answers the same question (online eval).
Watch the length correlation as a standing metric. A rising correlation between answer length and judge score means the rubric has drifted toward rewarding verbosity.
Escalate disagreements to humans. Cases where deterministic checks and the judge disagree, or where the judge is uncertain, are exactly the cases worth human review — and they become new golden cases (data flywheel).
Prod checklist
- Rubric with concrete, independently checkable criteria anchored to a reference answer
- Judge from a different model family than the generator; version pinned; temperature 0
- Structured per-criterion output with reasons
- Pairwise runs both orders; inconsistent results counted as ties and logged
- Calibration against 50–100 human labels; agreement, κ, false-pass and false-fail reported with every metric
- Recalibration on judge, rubric, or generator changes
- Length-bias correlation monitored
- Deterministic checks used wherever possible instead of the judge
- Human sign-off retained for high-stakes acceptance
- Judge disagreements routed to human review and into the golden set
Metrics to watch
| Metric | Why |
|---|---|
| Judge–human agreement and κ | Whether the scores mean anything |
| False-pass rate | Failures the judge is hiding |
| Score–length correlation | Length bias |
| Self-preference delta (judge on own vs other family) | Judge independence |
| Judge cost per eval run | Budget |
Related
- Golden dataset — reference answers
- Prompt evaluation — where judge scores are used
- Statistical hygiene — confidence intervals on judge scores
- Online eval — sampled production grading
- Reflection — critics need the same calibration