Retry ladder → Bounded escalation with visible failure

RAG Parsing Retry Ladder Explained: Escalate, Never Fail Silently

What it is

Primary parser → quality checks (production per-page heuristics)
  ├── passed → continue
  └── failed → secondary parser → checks
                 ├── passed → continue
                 └── failed → vision model (VLM) → checks
                                ├── passed → continue
                                └── failed → human review queue  (bounded; only if unresolved)

Rungs are ordered by cost and capability:

RungExampleCost/pageLatencyHandles
1 PrimaryDocling (digital) / PaddleOCR (scan)¢0.01–0.1ms–sMost pages
2 SecondaryDifferent engine (Tesseract, cloud OCR, MinerU, higher DPI)¢0.1–1sEngine-specific failures, bad DPI
3 VLMClaude/GPT-4o-class or self-hosted VLM: "transcribe this page as Markdown"¢1–10s–10sHard layouts, handwriting, stamps, degraded scans, tables
4 HumanReview UI with page image$hours–daysEverything else; high-value docs only

The ladder turns "parser failed" from an exception into a workflow with a budget and an audit trail.

How to implement

  1. Wrap each rung as parse(page, context) → (elements, stats); the validator (production per-page heuristics) decides pass/fail with a reason.
  2. Pass the failure reason downward: "low confidence, dictionary ratio 0.3" tells rung 2 to try higher DPI + a different engine; "reading order" tells it to try XY-cut; "table_structure" tells rung 3 to prompt for table transcription.
  3. Bound it: max rungs per page (3 automatic), max automatic cost per document, per-page timeout per rung, and a global daily budget for rung 3.
  4. Rung 3 (VLM) prompt: page image + "Transcribe all text in reading order as Markdown; tables as Markdown tables; mark unreadable regions as [unreadable]". Validate its output with the same checks (it can hallucinate fluently).
  5. Rung 4 gate: only for documents/sources marked human_review_eligible (value, compliance) — otherwise mark the page degraded and stop.
  6. Visible failure state: page gets status=degraded, reason=…, rungs_tried=[…]; document gets degraded_pages=n. The index stores the flag; the generation layer surfaces "part of this document could not be read" when a degraded document is cited; the ingest UI shows it.
  7. Log every rung: which rung succeeded, cost, latency, reason. This is your data for tuning thresholds and routing.

Small example

import time

class Rung:
    def __init__(self, name, fn, max_seconds, unit_cost): self.name, self.fn, self.max_seconds, self.unit_cost = name, fn, max_seconds, unit_cost

def run_ladder(page, rungs, validate, budget_cents, human_eligible=False):
    spent, trail = 0.0, []
    for rung in rungs:
        if spent + rung.unit_cost > budget_cents:
            trail.append({"rung": rung.name, "skipped": "budget"}); break
        t0 = time.time()
        try:
            elements = rung.fn(page, trail)                      # trail carries prior failure reasons
            verdict = validate(elements, page)
        except TimeoutError:
            verdict = {"decision": "retry", "reason": "timeout"}; elements = None
        dt = time.time() - t0; spent += rung.unit_cost
        trail.append({"rung": rung.name, "seconds": dt, "decision": verdict["decision"], "reason": verdict.get("reason")})
        if verdict["decision"] == "continue":
            return {"status": "ok", "elements": elements, "rung": rung.name, "cost_cents": spent, "trail": trail}
    if human_eligible:
        enqueue_human_review(page, trail)
        return {"status": "pending_human", "elements": None, "cost_cents": spent, "trail": trail}
    return {"status": "degraded", "elements": None, "cost_cents": spent, "trail": trail}   # visible, never silent

rungs = [Rung("docling", parse_docling, 30, 0.02),
         Rung("paddle_300dpi", parse_paddle_hi, 60, 0.3),
         Rung("vlm_transcribe", parse_vlm, 60, 5.0)]
result = run_ladder(page, rungs, validate_page, budget_cents=6.0, human_eligible=page["source"] in HIGH_VALUE)

Pros

  • Recovers most failures automatically at a cost proportional to difficulty.
  • Failures are explicit with reasons and an audit trail → debuggable, reportable, improvable.
  • Cost-bounded: the VLM never runs on the whole corpus by accident.
  • Human queue is targeted: only genuinely hard pages from sources that justify the expense.
  • Trail data drives routing improvements (per-page classification and routing): if a source always ends at rung 3, route it there directly.

Cons

  • Latency: a page that walks all rungs takes minutes; ingestion SLAs must account for it.
  • VLM hallucination: rung 3 produces fluent text that passes heuristics but isn't what's on the page; needs stricter validation and sampling.
  • Complexity: more code paths, more configuration, more things to monitor.
  • Human queues rot if nobody owns them; pending pages become a silent backlog.

When to use

  • Every ingestion pipeline with heterogeneous documents. A single parser with no fallback will silently degrade on some fraction of pages.

When NOT to use (parts of it)

  • Rung 4 for low-value, high-volume sources — mark degraded and move on.
  • Rung 3 without governance approval for sensitive images (data residency & retention) — stop at rung 2 for those tenants.

Production at scale

Async and page-level. Rungs 2–4 run on separate queues with their own workers and autoscaling; the primary path never waits. A document's index entry is published with the pages that passed; degraded/pending pages are added later (new version, document versioning) when they resolve. Users get most of the document quickly.

Budgets at three levels. Per page (max rungs, max cents), per document (max total), per day per source (rung-3 spend cap with alerting). Budget exhaustion is a recorded outcome (skipped: budget), never a crash.

Reason-driven rung selection. Don't run a fixed sequence blindly. Map failure reasons to strategies: confidence → higher DPI/different engine; reading_order → alternative ordering; table_structure → VLM table transcription; replacement_characters (digital) → OCR. This cuts wasted rungs.

Validate the VLM harder. Extra checks on rung-3 output: length plausibility vs OCR line count, presence of the [unreadable] marker when confidence was low, agreement with rung-1/2 text on the readable portions (fuzzy ratio). Sample rung-3 outputs to human review at a higher rate (5–10%).

Human review as a product. Queue with SLA, ownership, and a UI (page image + editable transcription + bbox tools). Reviewed pages become golden pages (test-time validation against a golden sample). Track backlog age; alert on breaches. If the backlog grows unboundedly, the eligibility rule is too broad.

Degraded state through the whole stack. Index: degraded_pages, degraded_reason on the document. Retrieval: no change (the readable parts are useful). Generation: when a cited document is degraded, append "Note: N pages of this document could not be processed" (refusal / no-answer path refusal/limits). Ingest UI: red flag with the trail. Monitoring: degraded-page rate per source.

Metrics tell you where to invest. Rung-success distribution per source: if 40% of a source's pages need rung 2, fix the primary route for that source. If rung 3 succeeds where rung 2 failed on tables, add a better table model to rung 2. The ladder is a diagnostic instrument, not just a fallback.

Idempotency. A retried page must produce the same chunk IDs on success regardless of the rung (IDs from doc_id + version + page + element index, not from parser internals), so republishing after human review replaces rather than duplicates.

Prod checklist

  • Rungs defined with cost, timeout, and reason-driven strategy selection
  • Per-page, per-document, per-day budgets; exhaustion recorded, not thrown
  • Async queues per rung; partial documents published early, completed later as new versions
  • Stricter validation + higher review sampling for VLM output
  • Human review eligibility rules; queue SLA, ownership, backlog alerts; reviewed pages → golden set
  • Degraded flag propagated to index, generation (user-visible note), ingest UI, dashboards
  • Trail logged per page: rungs, reasons, cost, latency
  • Chunk IDs stable across rungs

Metrics to watch

MetricWhy
Rung-success distribution per sourceRouting and parser investment
Degraded-page rate per sourceVisible failure
Rung-3 spend per day; budget skipsCost control
Human queue backlog ageRot prevention
Rung-3 hallucination rate (review sample)Trust in VLM rung