OCR / digitization → Complex layout (multi-column, forms → layout model)

Layout Aware OCR for RAG: Fix Multi-Column Pages and Forms

What it is

Plain OCR returns text lines in whatever order the detector found them — typically top-to-bottom by y-coordinate, which for a two-column page means line 1 of column A, line 1 of column B, line 2 of column A… The result is unreadable and un-embeddable.

A layout analysis model (PaddleOCR PP-Structure / PP-DocLayout, Docling's layout model, LayoutLM-family, Surya layout) first segments the page into typed regions — text, title, list, table, figure, header, footer, key-value — with bounding boxes. Then reading order is inferred over regions (column detection, XY-cut, or a learned order model), and OCR/extraction is run per region. Tables get a dedicated table-structure model; forms get key-value pairing.

The same route applies to born-digital pages with complex layout (digital document already uses Docling's layout model); the difference here is the text comes from OCR.

How to implement

  1. Rasterise the page (300 dpi) as in scanned page.
  2. Run the layout model to get typed regions with bboxes and confidence.
  3. Order regions: detect columns by clustering region x-centres; within each column sort by y; read columns left-to-right (or per language direction). Use the engine's built-in ordering if it has one (PP-Structure recovery, Docling reading-order model) and verify.
  4. Per-region processing: text/title/list → OCR lines, join into paragraph; table → table-structure model → HTML/Markdown (tables); figure → VLM caption (charts, images, diagrams); header/footer → metadata, dropped from text (headers / footers / boilerplate stripping); key-value → pair label and value by geometry.
  5. Forms specifically: run a key-value extraction (PP-Structure KIE, LayoutLMv3, or a VLM with a schema) to emit {field: value} pairs; store them as structured metadata and as text ("Invoice number: 12345") so both filters and embeddings work.
  6. Emit common-schema elements with region type, bbox, page, confidence, parser=pp-structure@version.
  7. Validate: reading-order continuity, region coverage (are large parts of the page unassigned?), table structure sanity (production per-page heuristics).

Small example

import fitz, numpy as np
from paddleocr import PPStructure

engine = PPStructure(layout=True, table=True, ocr=True, show_log=False, use_gpu=True, recovery=True)

def parse_complex_page(page: fitz.Page, page_no: int) -> list[dict]:
    pix = page.get_pixmap(dpi=300, colorspace=fitz.csRGB)
    img = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
    regions = engine(img)                                   # [{type, bbox, res, ...}]
    # column-aware order: cluster by x-centre, then y within column
    def col(r): return round(((r["bbox"][0] + r["bbox"][2]) / 2) / (pix.width / 2.5))
    regions.sort(key=lambda r: (col(r), r["bbox"][1]))
    out = []
    for r in regions:
        base = {"page": page_no, "bbox": r["bbox"], "region_type": r["type"], "parser": "pp-structure@2.8"}
        if r["type"] == "table":
            out.append(base | {"type": "table", "html": r["res"]["html"], "text": html_to_markdown(r["res"]["html"])})
        elif r["type"] == "figure":
            out.append(base | {"type": "figure", "image_ref": save_crop(img, r["bbox"], page_no)})
        elif r["type"] in ("header", "footer"):
            continue
        else:
            text = " ".join(l["text"] for l in r["res"]) if isinstance(r["res"], list) else ""
            conf = float(np.mean([l["confidence"] for l in r["res"]])) if isinstance(r["res"], list) and r["res"] else 0
            out.append(base | {"type": "title" if r["type"] == "title" else "paragraph", "text": text, "confidence": conf})
    return out

Pros

  • Correct reading order on multi-column pages — the difference between usable and unusable text.
  • Typed regions give you headings, tables, and figures for free → layout-aware chunking (layout / structure-aware).
  • Forms become structured data: filters on invoice number, date, amount.
  • Headers/footers identified by the model, not by heuristics.

Cons

  • Another model with its own errors: missed regions, merged columns, tables detected as text.
  • Slower than plain OCR (layout pass + per-region OCR + table model).
  • Reading-order models are template-sensitive; unusual designs (magazines, brochures, sideways tables) still fail.
  • Forms are hard: checkboxes, handwriting, multi-page forms, labels far from values.
  • Region coverage gaps silently drop text.

When to use

  • Pages classified as complex layout (per-page classification and routing): ≥2 columns, forms, invoices, statements, scientific papers, newsletters.
  • Any scanned page that contains a table.

When NOT to use

  • Simple single-column scans — plain OCR (scanned page) is faster and the layout model adds little.
  • Born-digital pages — use the native-layer parser's layout model (digital document) instead of OCR.
  • Pages that are essentially one figure (charts, images, diagrams).

Production at scale

Region coverage check. Sum region areas vs page text area (from a quick full-page OCR or from the detector's raw line boxes). If a large fraction of detected text lines fall outside any region, the layout model missed something → flag and escalate (bounded escalation with visible failure). This is the most common silent failure.

Reading-order validation. Sentence-continuity check across region boundaries; for two-column pages, check that column A's last line and column B's first line don't read as one sentence if they were ordered that way. Cheap, catches interleaving.

Forms: schema-driven extraction. Don't rely on generic KIE for known form types. Maintain a per-template schema (fields, expected types, validation regexes) and validate extracted values; failures route to a VLM with the schema in the prompt, then to human review. Store form values as structured metadata (filterable) and render them into text for embedding.

Tables from scans. Table-structure models (TableFormer, SLANet) work on clean tables and fail on borderless or merged-cell tables. Validate: consistent column count per row, numeric columns actually numeric, header row present. Failing tables → VLM with "transcribe this table as Markdown" (charts, images, diagrams), which is often better than a small table model on hard cases.

Cost tiers. Layout + OCR + table model on GPU is 2–5× the cost of plain OCR per page. Route only genuinely complex pages here (per-page classification and routing); log the fraction per source.

Template libraries. Enterprises have recurring templates (the same invoice layout 50k times). Detect template by layout fingerprint (region-type histogram + positions) and apply template-specific rules/ROIs. Big accuracy and speed win for high-volume forms.

Multi-page tables and forms. Detect continuation (same column x-positions on consecutive pages, "continued" text) and merge at reassembly; store table_id spanning pages.

Human-review integration. Form extraction with low confidence or failed validation goes to a review UI showing the page image with bboxes; corrections feed back as training/golden data (data flywheel).

Versioning. Layout model weights and table model weights are separate versions; store both. A layout model upgrade changes region boundaries → chunks → re-index for pages of this class only (filter by page_class).

Prod checklist

  • Layout model + column-aware ordering; engine's recovery/order used and verified
  • Per-region processing: text/table/figure/header-footer/key-value
  • Region coverage and reading-order checks → retry ladder
  • Form schemas per template; value validation; structured + text output
  • Table validation; VLM fallback for hard tables
  • Template fingerprinting for high-volume forms
  • Multi-page table/form continuation at reassembly
  • Human-review UI with page image + bboxes; corrections captured
  • Layout and table model versions stored per element

Metrics to watch

MetricWhy
Region coverage ratio per pageMissed content
Reading-order failure rateInterleaving
Form field validation pass rate per templateExtraction quality
Table structure validation pass rateTable quality
$/page vs plain OCRRouting discipline