Validation → Production per-page heuristics (no ground truth)

Parse Quality Checks Without Ground Truth: Production Heuristics

What it is

The golden-sample evaluation (test-time validation against a golden sample) tells you how good the parser is on average. It cannot tell you that this page, from a new scanner, just came out as ���� ���� ���� or as an empty string. Production validation is a set of per-page heuristic checks that need no ground truth:

CheckSignalCatches
minimum_textchars/words per page vs expectations for the page classEmpty output, failed OCR, wrong route
replacement_charactersratio of U+FFFD and private-use charsFont-map failures, encoding errors
control_charactersnon-printable ratioBinary/garbage decode
dictionary_ratiofraction of tokens in a vocabulary (per language)OCR garbage with high confidence
reading_ordersentence-continuity score (reading order)Column interleaving
element_ids / coordinatesevery element has a unique ID and a valid bbox within page boundsAdapter bugs, coordinate errors
table_structurerectangular grid, header present, totals reconcile (tables)Table model failures
confidenceparser/OCR mean and low-confidence ratioBad scans
stripped_fractionboilerplate removed vs total (headers / footers / boilerplate stripping)Over-stripping
heading_countheadings per page/doc vs source baselineStructure regression

Each check yields pass/warn/fail; the page's worst result decides whether it continues, retries (bounded escalation with visible failure), or is flagged. Everything is logged so degradation is visible.

How to implement

  1. Define checks as pure functions over the common element schema; each returns (status, score, detail).
  2. Thresholds per page class and source, learned from the golden sample and from production distributions (e.g. minimum_text for a scanned invoice is 50 words; for a digital report page 150).
  3. Run inline after parsing each page; cost is microseconds (dictionary lookup is the heaviest).
  4. Decision policy: any fail → retry ladder; ≥2 warn → retry ladder; else continue. Record all results on the page/document (validation: {...}).
  5. Never emit empty text silently. A page with no text and no failure is a bug; force a fail on empty output unless the page class is blank and the blank classifier agrees.
  6. Aggregate to document level: fraction of failed pages; > X% → document-level flag (and don't index a document whose core pages failed).
  7. Dashboards and alerts per source: failure rate by check; sudden shifts are the earliest signal of a source or parser change.

Small example

import re, unicodedata

WORD = re.compile(r"[A-Za-z]{2,}")
PUA = lambda ch: 0xE000 <= ord(ch) <= 0xF8FF

def check_text_quality(text: str, vocab: set, min_words: int) -> dict:
    words = WORD.findall(text)
    n = len(text) or 1
    repl = sum(ch == "�" or PUA(ch) for ch in text) / n
    ctrl = sum(unicodedata.category(ch)[0] == "C" and ch not in "\n\t\r" for ch in text) / n
    dict_ratio = (sum(w.lower() in vocab for w in words) / len(words)) if words else 0.0
    status = "pass"
    if len(words) < min_words: status = "fail"
    elif repl > 0.01 or ctrl > 0.01: status = "fail"
    elif dict_ratio < 0.6: status = "warn" if dict_ratio > 0.4 else "fail"
    return {"check": "text_quality", "status": status,
            "detail": {"words": len(words), "replacement_ratio": repl, "control_ratio": ctrl, "dictionary_ratio": dict_ratio}}

def check_coordinates(elements, page_size) -> dict:
    W, H = page_size; bad = 0; ids = set()
    for e in elements:
        x0, y0, x1, y1 = e["bbox"]
        if not (0 <= x0 < x1 <= W + 1 and 0 <= y0 < y1 <= H + 1): bad += 1
        if e["id"] in ids: bad += 1
        ids.add(e["id"])
    return {"check": "coordinates", "status": "fail" if bad else "pass", "detail": {"bad": bad, "n": len(elements)}}

def check_confidence(elements, low=0.7) -> dict:
    confs = [e["confidence"] for e in elements if e.get("confidence") is not None]
    if not confs: return {"check": "confidence", "status": "pass", "detail": {"n": 0}}
    low_ratio = sum(c < low for c in confs) / len(confs); mean = sum(confs) / len(confs)
    status = "fail" if low_ratio > 0.3 or mean < 0.6 else "warn" if low_ratio > 0.15 else "pass"
    return {"check": "confidence", "status": status, "detail": {"mean": mean, "low_ratio": low_ratio}}

def validate_page(page, cfg, vocab):
    results = [check_text_quality(page["text"], vocab, cfg["min_words"][page["page_class"]]),
               check_coordinates(page["elements"], page["page_size"]),
               check_confidence(page["elements"]),
               check_reading_order(page["elements"]),          # from reading order
               *[check_table(t) for t in page.get("tables", [])]]  # from tables
    worst = max((r["status"] for r in results), key=["pass", "warn", "fail"].index)
    warns = sum(r["status"] == "warn" for r in results)
    decision = "retry" if worst == "fail" or warns >= 2 else "continue"
    return {"decision": decision, "results": results}

Pros

  • Catches silent failures (empty pages, garbage decode, wrong route) that would otherwise produce confident wrong answers.
  • Nearly free to run.
  • Feeds the retry ladder with a reason, so escalation is targeted.
  • Early warning for source or parser changes via dashboards.
  • Produces the candidates for the golden set (test-time validation against a golden sample): failed pages get reviewed and labelled.

Cons

  • Heuristics have false positives (a page of part numbers has a low dictionary ratio; a legitimately blank page fails minimum_text) and false negatives (fluent OCR errors pass every check).
  • Thresholds need tuning per class/source, and drift as sources change.
  • Doesn't measure correctness, only plausibility.
  • Language-dependent checks need per-language vocabularies.

When to use

Always, on every page, in every environment. This is the production safety net for the whole ingestion pipeline.

When NOT to use (specific checks)

  • Dictionary-ratio on code, part lists, or multilingual pages — replace with character-class entropy or per-language vocab detection.
  • minimum_text on pages the classifier already labelled blank/figure — use class-specific thresholds.

Production at scale

Silent degradation is the failure to design against. The source file says it: a page that silently degrades to empty text produces a confident wrong answer instead of an error. So: empty output without a recorded failure is itself a fail; a document whose critical pages failed is not indexed (or indexed with a degraded=true flag that the generation layer surfaces to the user); and every decision is logged.

Thresholds as versioned config, per (source, page_class). Bootstrap from the golden sample distributions (e.g. 5th percentile of words per page for that class), then tune from production. Changing thresholds is a change that can re-route thousands of pages — review the projected impact before deploying (dry-run mode that logs decisions without acting).

Document-level aggregation. Store validation_summary on the document: pages checked, failed, warned, retried, escalated, final degraded flag. Expose it in the ingest UI so a user who asks "why is my document not answering" gets a real answer.

Dashboards per source and check. Failure rate by check is the most useful ingestion chart you will have. A jump in replacement_characters for one source = new export tool with a font-map problem. A jump in minimum_text = scanner change. A jump in reading_order after a deploy = parser regression.

Language handling. Detect language per page (fastText/lingua) before the dictionary check; keep vocabularies per language; skip the check for unsupported languages and record check_skipped.

Confidence calibration. OCR confidence thresholds should be calibrated on the golden sample per engine version (scanned page); recalibrate on upgrade.

Sampling for human review. Even passing pages get sampled (0.1–1%) into a review queue; that's how you find fluent-but-wrong OCR that heuristics can't catch, and how the golden set grows (data flywheel).

Cost of retries. Validation drives the retry ladder; a too-strict threshold sends 20% of pages to the expensive rungs. Track retry rate per check and the success rate of each rung; a check that triggers retries which then pass unchanged is miscalibrated.

Prod checklist

  • Checks implemented over the common element schema; pure and cheap
  • Thresholds per (source, page_class) in versioned config; dry-run mode for changes
  • Empty output is a fail unless classified blank; degraded documents not indexed silently
  • Decision + results stored per page; summary per document; visible in ingest UI
  • Failure-rate dashboards by (source, check); alerts on shifts
  • Language detection before dictionary checks; per-language vocabularies
  • Confidence thresholds calibrated per engine version
  • Random sample of passing pages to human review
  • Retry rate and rung success rate tracked per check

Metrics to watch

MetricWhy
Failure rate by (source, check)Source/parser change detection
Retry rate; rung success rateThreshold calibration
Degraded documents indexed / not indexedVisibility of failure
Passing-page review findingsHeuristic blind spots