Validation → Production per-page heuristics (no ground truth)
What it is
The golden-sample evaluation (test-time validation against a golden sample) tells you how good the parser is on average. It cannot tell you that this page, from a new scanner, just came out as ���� ���� ���� or as an empty string. Production validation is a set of per-page heuristic checks that need no ground truth:
| Check | Signal | Catches |
|---|---|---|
minimum_text | chars/words per page vs expectations for the page class | Empty output, failed OCR, wrong route |
replacement_characters | ratio of U+FFFD and private-use chars | Font-map failures, encoding errors |
control_characters | non-printable ratio | Binary/garbage decode |
dictionary_ratio | fraction of tokens in a vocabulary (per language) | OCR garbage with high confidence |
reading_order | sentence-continuity score (reading order) | Column interleaving |
element_ids / coordinates | every element has a unique ID and a valid bbox within page bounds | Adapter bugs, coordinate errors |
table_structure | rectangular grid, header present, totals reconcile (tables) | Table model failures |
confidence | parser/OCR mean and low-confidence ratio | Bad scans |
stripped_fraction | boilerplate removed vs total (headers / footers / boilerplate stripping) | Over-stripping |
heading_count | headings per page/doc vs source baseline | Structure regression |
Each check yields pass/warn/fail; the page's worst result decides whether it continues, retries (bounded escalation with visible failure), or is flagged. Everything is logged so degradation is visible.
How to implement
- Define checks as pure functions over the common element schema; each returns
(status, score, detail). - Thresholds per page class and source, learned from the golden sample and from production distributions (e.g.
minimum_textfor a scanned invoice is 50 words; for a digital report page 150). - Run inline after parsing each page; cost is microseconds (dictionary lookup is the heaviest).
- Decision policy: any
fail→ retry ladder; ≥2warn→ retry ladder; else continue. Record all results on the page/document (validation: {...}). - Never emit empty text silently. A page with no text and no failure is a bug; force a
failon empty output unless the page class isblankand the blank classifier agrees. - Aggregate to document level: fraction of failed pages; > X% → document-level flag (and don't index a document whose core pages failed).
- Dashboards and alerts per source: failure rate by check; sudden shifts are the earliest signal of a source or parser change.
Small example
import re, unicodedata
WORD = re.compile(r"[A-Za-z]{2,}")
PUA = lambda ch: 0xE000 <= ord(ch) <= 0xF8FF
def check_text_quality(text: str, vocab: set, min_words: int) -> dict:
words = WORD.findall(text)
n = len(text) or 1
repl = sum(ch == "�" or PUA(ch) for ch in text) / n
ctrl = sum(unicodedata.category(ch)[0] == "C" and ch not in "\n\t\r" for ch in text) / n
dict_ratio = (sum(w.lower() in vocab for w in words) / len(words)) if words else 0.0
status = "pass"
if len(words) < min_words: status = "fail"
elif repl > 0.01 or ctrl > 0.01: status = "fail"
elif dict_ratio < 0.6: status = "warn" if dict_ratio > 0.4 else "fail"
return {"check": "text_quality", "status": status,
"detail": {"words": len(words), "replacement_ratio": repl, "control_ratio": ctrl, "dictionary_ratio": dict_ratio}}
def check_coordinates(elements, page_size) -> dict:
W, H = page_size; bad = 0; ids = set()
for e in elements:
x0, y0, x1, y1 = e["bbox"]
if not (0 <= x0 < x1 <= W + 1 and 0 <= y0 < y1 <= H + 1): bad += 1
if e["id"] in ids: bad += 1
ids.add(e["id"])
return {"check": "coordinates", "status": "fail" if bad else "pass", "detail": {"bad": bad, "n": len(elements)}}
def check_confidence(elements, low=0.7) -> dict:
confs = [e["confidence"] for e in elements if e.get("confidence") is not None]
if not confs: return {"check": "confidence", "status": "pass", "detail": {"n": 0}}
low_ratio = sum(c < low for c in confs) / len(confs); mean = sum(confs) / len(confs)
status = "fail" if low_ratio > 0.3 or mean < 0.6 else "warn" if low_ratio > 0.15 else "pass"
return {"check": "confidence", "status": status, "detail": {"mean": mean, "low_ratio": low_ratio}}
def validate_page(page, cfg, vocab):
results = [check_text_quality(page["text"], vocab, cfg["min_words"][page["page_class"]]),
check_coordinates(page["elements"], page["page_size"]),
check_confidence(page["elements"]),
check_reading_order(page["elements"]), # from reading order
*[check_table(t) for t in page.get("tables", [])]] # from tables
worst = max((r["status"] for r in results), key=["pass", "warn", "fail"].index)
warns = sum(r["status"] == "warn" for r in results)
decision = "retry" if worst == "fail" or warns >= 2 else "continue"
return {"decision": decision, "results": results}
Pros
- Catches silent failures (empty pages, garbage decode, wrong route) that would otherwise produce confident wrong answers.
- Nearly free to run.
- Feeds the retry ladder with a reason, so escalation is targeted.
- Early warning for source or parser changes via dashboards.
- Produces the candidates for the golden set (test-time validation against a golden sample): failed pages get reviewed and labelled.
Cons
- Heuristics have false positives (a page of part numbers has a low dictionary ratio; a legitimately blank page fails
minimum_text) and false negatives (fluent OCR errors pass every check). - Thresholds need tuning per class/source, and drift as sources change.
- Doesn't measure correctness, only plausibility.
- Language-dependent checks need per-language vocabularies.
When to use
Always, on every page, in every environment. This is the production safety net for the whole ingestion pipeline.
When NOT to use (specific checks)
- Dictionary-ratio on code, part lists, or multilingual pages — replace with character-class entropy or per-language vocab detection.
minimum_texton pages the classifier already labelled blank/figure — use class-specific thresholds.
Production at scale
Silent degradation is the failure to design against. The source file says it: a page that silently degrades to empty text produces a confident wrong answer instead of an error. So: empty output without a recorded failure is itself a fail; a document whose critical pages failed is not indexed (or indexed with a degraded=true flag that the generation layer surfaces to the user); and every decision is logged.
Thresholds as versioned config, per (source, page_class). Bootstrap from the golden sample distributions (e.g. 5th percentile of words per page for that class), then tune from production. Changing thresholds is a change that can re-route thousands of pages — review the projected impact before deploying (dry-run mode that logs decisions without acting).
Document-level aggregation. Store validation_summary on the document: pages checked, failed, warned, retried, escalated, final degraded flag. Expose it in the ingest UI so a user who asks "why is my document not answering" gets a real answer.
Dashboards per source and check. Failure rate by check is the most useful ingestion chart you will have. A jump in replacement_characters for one source = new export tool with a font-map problem. A jump in minimum_text = scanner change. A jump in reading_order after a deploy = parser regression.
Language handling. Detect language per page (fastText/lingua) before the dictionary check; keep vocabularies per language; skip the check for unsupported languages and record check_skipped.
Confidence calibration. OCR confidence thresholds should be calibrated on the golden sample per engine version (scanned page); recalibrate on upgrade.
Sampling for human review. Even passing pages get sampled (0.1–1%) into a review queue; that's how you find fluent-but-wrong OCR that heuristics can't catch, and how the golden set grows (data flywheel).
Cost of retries. Validation drives the retry ladder; a too-strict threshold sends 20% of pages to the expensive rungs. Track retry rate per check and the success rate of each rung; a check that triggers retries which then pass unchanged is miscalibrated.
Prod checklist
- Checks implemented over the common element schema; pure and cheap
- Thresholds per (source, page_class) in versioned config; dry-run mode for changes
- Empty output is a fail unless classified blank; degraded documents not indexed silently
- Decision + results stored per page; summary per document; visible in ingest UI
- Failure-rate dashboards by (source, check); alerts on shifts
- Language detection before dictionary checks; per-language vocabularies
- Confidence thresholds calibrated per engine version
- Random sample of passing pages to human review
- Retry rate and rung success rate tracked per check
Metrics to watch
| Metric | Why |
|---|---|
| Failure rate by (source, check) | Source/parser change detection |
| Retry rate; rung success rate | Threshold calibration |
| Degraded documents indexed / not indexed | Visibility of failure |
| Passing-page review findings | Heuristic blind spots |
Related
- Golden-sample validation — thresholds come from there
- Retry ladder — consumer of decisions
- Reading order, Tables — specific checks
- Routing — mis-route detection
- Data flywheel