Validation → Test-time validation against a golden sample

RAG Parser Evaluation: Golden Sample Testing in CI

What it is

A golden sample is a set of pages from your corpus (50–300, stratified by source and page class) with ground truth produced or verified by humans: the correct text, the correct fields, the correct table cells, the correct reading order. The parser's output on those pages is compared to ground truth with standard metrics:

MetricWhat it measuresTypical target
CER / WER (character/word error rate)OCR/text fidelity: edit distance ÷ reference lengthCER < 2% clean scans; < 0.5% digital
Field-level P/R/F1Extraction of named fields (invoice no., date, total, party names)F1 > 0.95 on key fields
Table-cell accuracy / TEDSCell content + structure (Tree Edit Distance Similarity)TEDS > 0.9
Reading-order accuracyKendall tau / pairwise agreement over block order> 0.95
Heading detection P/RSection headers found, levels rightP/R > 0.9
Formula exact-match / edit distance (if math)LaTeX fidelityDomain-dependent

This is "test time": run offline, with ground truth, on parser changes. Production has no ground truth — that's production per-page heuristics.

How to implement

  1. Sample stratified: by source, page class (digital/scanned/complex/formula/figure), template, language. Include the ugly ones (faxes, rotated, handwritten annotations) in proportion to their real share plus a few extra.
  2. Produce ground truth: for digital pages, start from the best parser output and correct; for scans, transcribe or correct OCR in a review tool with the page image; for tables, enter the cell grid; for fields, annotate {field: value, bbox}. Two annotators on a subset; measure agreement.
  3. Store as a versioned dataset (golden/parsing/v3/…) with page images, ground-truth JSON, and metadata (source, class, template). Never in the vector store; in git-LFS or object storage with a manifest.
  4. Compute metrics per page and aggregate per stratum; report distributions, not just means.
  5. Gate parser changes: a parser or model-weights upgrade must not regress any stratum beyond a tolerance; block the rollout otherwise (the testing pyramid).
  6. Grow from production failures: every parser incident found via production per-page heuristics or user reports becomes a golden page (data flywheel).

Small example

import json, jiwer
from apted import APTED
from apted.helpers import Tree

def cer(ref: str, hyp: str) -> float:
    return jiwer.cer(ref, hyp)

def wer(ref: str, hyp: str) -> float:
    return jiwer.wer(ref, hyp)

def field_prf(gt: dict, pred: dict, norm=lambda s: " ".join(str(s).lower().split())):
    tp = sum(1 for k, v in gt.items() if k in pred and norm(pred[k]) == norm(v))
    fp = sum(1 for k in pred if k not in gt or norm(pred[k]) != norm(gt.get(k, "")))
    fn = sum(1 for k in gt if k not in pred or norm(pred[k]) != norm(gt[k]))
    p = tp / (tp + fp) if tp + fp else 0.0; r = tp / (tp + fn) if tp + fn else 0.0
    return {"precision": p, "recall": r, "f1": 2 * p * r / (p + r) if p + r else 0.0}

def cell_accuracy(gt_cells: list[list[str]], pred_cells: list[list[str]]) -> float:
    if not gt_cells: return 0.0
    total = sum(len(r) for r in gt_cells); correct = 0
    for i, row in enumerate(gt_cells):
        for j, v in enumerate(row):
            try: correct += (pred_cells[i][j].strip() == v.strip())
            except IndexError: pass
    return correct / total

def reading_order_tau(gt_ids: list[str], pred_ids: list[str]) -> float:
    pos = {b: i for i, b in enumerate(pred_ids)}
    pairs = [(a, b) for i, a in enumerate(gt_ids) for b in gt_ids[i + 1:] if a in pos and b in pos]
    if not pairs: return 0.0
    agree = sum(1 for a, b in pairs if pos[a] < pos[b])
    return (2 * agree - len(pairs)) / len(pairs)          # Kendall tau in [-1, 1]

def evaluate(golden_dir, parser):
    rows = []
    for page in json.load(open(f"{golden_dir}/manifest.json")):
        gt = json.load(open(page["gt"])); out = parser(page["image"])
        rows.append({"id": page["id"], "class": page["page_class"], "source": page["source"],
                     "cer": cer(gt["text"], out["text"]), "wer": wer(gt["text"], out["text"]),
                     **field_prf(gt.get("fields", {}), out.get("fields", {})),
                     "cell_acc": cell_accuracy(gt.get("cells", []), out.get("cells", [])),
                     "order_tau": reading_order_tau(gt["block_order"], out["block_order"])})
    return rows   # → pandas groupby(class, source).describe()

Pros

  • Objective parser selection and upgrade gating instead of "looks fine on three PDFs".
  • Per-stratum visibility: you learn which sources/templates the parser fails on.
  • Root-cause attribution: when end-to-end RAG quality drops, parsing metrics tell you whether it's the parser.
  • Golden pages double as regression tests and as training data for parser fine-tuning.

Cons

  • Annotation cost: table cells and reading order are slow to label; 200 pages can be person-weeks.
  • Sample drift: the corpus changes; the golden set must be refreshed or it stops representing production.
  • Metric limitations: CER treats a wrong digit in a total the same as a wrong letter in a footnote; you need field-level and downstream QA metrics to weight what matters.
  • Doesn't catch production-only failures (new templates) — that's production per-page heuristics's job.

When to use

  • Before choosing a parser for a new source type.
  • Before every parser/model-weights/config upgrade.
  • When end-to-end evals regress and you need to localise the cause (component vs end-to-end).

When NOT to use (as the only validation)

Production at scale

Stratify by what breaks. The mean CER over 200 random pages hides a 30% CER on the 5% of pages from the fax source. Report per (source, page_class, template) and gate on the worst stratum that matters.

Weight by importance. Define critical fields (amounts, dates, IDs, party names) and gate on their F1 separately. A parser with 0.3% CER that flips one digit in totals is worse than 1% CER on prose.

Downstream metric in the same run. Alongside parsing metrics, run the retrieval and QA golden set (retrieval evals, generation evals) on documents parsed with the candidate parser. The final gate is "does end-to-end answer accuracy hold", not only "is CER lower".

CI integration. Parser evaluation runs in CI on every change to parser code, model weights, routing thresholds, or boilerplate rules. Fail the build on regression beyond tolerance (e.g. CER +0.5 pts, cell accuracy −2 pts, order tau −0.02, critical-field F1 −1 pt) in any gated stratum (the testing pyramid).

Annotation tooling. A review UI showing page image + parser output side by side, with click-to-correct; export ground truth JSON. The same UI serves the human-review rung of the retry ladder (bounded escalation with visible failure), so annotations accumulate from operations, not just from a one-off labelling project.

Inter-annotator agreement. Measure on a 10% overlap; if agreement is low on reading order or table structure, the guidelines are ambiguous — fix them before trusting the metrics.

Refresh policy. Add N pages per month from production failures and from new templates/sources; retire pages whose sources are gone. Version the golden set; tie each parser eval result to a golden-set version.

Cost of running. Parsing 300 pages with all routes takes minutes on one GPU; it's cheap. What's expensive is neglecting it and finding out in production that a Docling upgrade dropped every heading in the HR handbook.

Prod checklist

  • Stratified golden sample with page images, GT text/fields/cells/order; versioned
  • Metrics per stratum: CER/WER, field P/R/F1 (critical fields separately), cell accuracy/TEDS, order tau, heading P/R
  • Downstream retrieval/QA golden run with the candidate parser
  • CI gate with per-stratum tolerances
  • Annotation/review UI shared with the retry-ladder human queue
  • Inter-annotator agreement measured
  • Monthly refresh from production failures; retire stale pages

Metrics to watch

MetricWhy
CER/WER per stratumText fidelity
Critical-field F1What actually matters
Table cell accuracy / TEDSTable questions
Reading-order tauSentence integrity
Downstream QA accuracy with candidate parserThe real gate