OCR / digitization → Complex layout (multi-column, forms → layout model)
What it is
Plain OCR returns text lines in whatever order the detector found them — typically top-to-bottom by y-coordinate, which for a two-column page means line 1 of column A, line 1 of column B, line 2 of column A… The result is unreadable and un-embeddable.
A layout analysis model (PaddleOCR PP-Structure / PP-DocLayout, Docling's layout model, LayoutLM-family, Surya layout) first segments the page into typed regions — text, title, list, table, figure, header, footer, key-value — with bounding boxes. Then reading order is inferred over regions (column detection, XY-cut, or a learned order model), and OCR/extraction is run per region. Tables get a dedicated table-structure model; forms get key-value pairing.
The same route applies to born-digital pages with complex layout (digital document already uses Docling's layout model); the difference here is the text comes from OCR.
How to implement
- Rasterise the page (300 dpi) as in scanned page.
- Run the layout model to get typed regions with bboxes and confidence.
- Order regions: detect columns by clustering region x-centres; within each column sort by y; read columns left-to-right (or per language direction). Use the engine's built-in ordering if it has one (PP-Structure
recovery, Docling reading-order model) and verify. - Per-region processing:
text/title/list→ OCR lines, join into paragraph;table→ table-structure model → HTML/Markdown (tables);figure→ VLM caption (charts, images, diagrams);header/footer→ metadata, dropped from text (headers / footers / boilerplate stripping);key-value→ pair label and value by geometry. - Forms specifically: run a key-value extraction (PP-Structure KIE, LayoutLMv3, or a VLM with a schema) to emit
{field: value}pairs; store them as structured metadata and as text ("Invoice number: 12345") so both filters and embeddings work. - Emit common-schema elements with region type, bbox, page, confidence,
parser=pp-structure@version. - Validate: reading-order continuity, region coverage (are large parts of the page unassigned?), table structure sanity (production per-page heuristics).
Small example
import fitz, numpy as np
from paddleocr import PPStructure
engine = PPStructure(layout=True, table=True, ocr=True, show_log=False, use_gpu=True, recovery=True)
def parse_complex_page(page: fitz.Page, page_no: int) -> list[dict]:
pix = page.get_pixmap(dpi=300, colorspace=fitz.csRGB)
img = np.frombuffer(pix.samples, np.uint8).reshape(pix.height, pix.width, 3)
regions = engine(img) # [{type, bbox, res, ...}]
# column-aware order: cluster by x-centre, then y within column
def col(r): return round(((r["bbox"][0] + r["bbox"][2]) / 2) / (pix.width / 2.5))
regions.sort(key=lambda r: (col(r), r["bbox"][1]))
out = []
for r in regions:
base = {"page": page_no, "bbox": r["bbox"], "region_type": r["type"], "parser": "pp-structure@2.8"}
if r["type"] == "table":
out.append(base | {"type": "table", "html": r["res"]["html"], "text": html_to_markdown(r["res"]["html"])})
elif r["type"] == "figure":
out.append(base | {"type": "figure", "image_ref": save_crop(img, r["bbox"], page_no)})
elif r["type"] in ("header", "footer"):
continue
else:
text = " ".join(l["text"] for l in r["res"]) if isinstance(r["res"], list) else ""
conf = float(np.mean([l["confidence"] for l in r["res"]])) if isinstance(r["res"], list) and r["res"] else 0
out.append(base | {"type": "title" if r["type"] == "title" else "paragraph", "text": text, "confidence": conf})
return out
Pros
- Correct reading order on multi-column pages — the difference between usable and unusable text.
- Typed regions give you headings, tables, and figures for free → layout-aware chunking (layout / structure-aware).
- Forms become structured data: filters on invoice number, date, amount.
- Headers/footers identified by the model, not by heuristics.
Cons
- Another model with its own errors: missed regions, merged columns, tables detected as text.
- Slower than plain OCR (layout pass + per-region OCR + table model).
- Reading-order models are template-sensitive; unusual designs (magazines, brochures, sideways tables) still fail.
- Forms are hard: checkboxes, handwriting, multi-page forms, labels far from values.
- Region coverage gaps silently drop text.
When to use
- Pages classified as complex layout (per-page classification and routing): ≥2 columns, forms, invoices, statements, scientific papers, newsletters.
- Any scanned page that contains a table.
When NOT to use
- Simple single-column scans — plain OCR (scanned page) is faster and the layout model adds little.
- Born-digital pages — use the native-layer parser's layout model (digital document) instead of OCR.
- Pages that are essentially one figure (charts, images, diagrams).
Production at scale
Region coverage check. Sum region areas vs page text area (from a quick full-page OCR or from the detector's raw line boxes). If a large fraction of detected text lines fall outside any region, the layout model missed something → flag and escalate (bounded escalation with visible failure). This is the most common silent failure.
Reading-order validation. Sentence-continuity check across region boundaries; for two-column pages, check that column A's last line and column B's first line don't read as one sentence if they were ordered that way. Cheap, catches interleaving.
Forms: schema-driven extraction. Don't rely on generic KIE for known form types. Maintain a per-template schema (fields, expected types, validation regexes) and validate extracted values; failures route to a VLM with the schema in the prompt, then to human review. Store form values as structured metadata (filterable) and render them into text for embedding.
Tables from scans. Table-structure models (TableFormer, SLANet) work on clean tables and fail on borderless or merged-cell tables. Validate: consistent column count per row, numeric columns actually numeric, header row present. Failing tables → VLM with "transcribe this table as Markdown" (charts, images, diagrams), which is often better than a small table model on hard cases.
Cost tiers. Layout + OCR + table model on GPU is 2–5× the cost of plain OCR per page. Route only genuinely complex pages here (per-page classification and routing); log the fraction per source.
Template libraries. Enterprises have recurring templates (the same invoice layout 50k times). Detect template by layout fingerprint (region-type histogram + positions) and apply template-specific rules/ROIs. Big accuracy and speed win for high-volume forms.
Multi-page tables and forms. Detect continuation (same column x-positions on consecutive pages, "continued" text) and merge at reassembly; store table_id spanning pages.
Human-review integration. Form extraction with low confidence or failed validation goes to a review UI showing the page image with bboxes; corrections feed back as training/golden data (data flywheel).
Versioning. Layout model weights and table model weights are separate versions; store both. A layout model upgrade changes region boundaries → chunks → re-index for pages of this class only (filter by page_class).
Prod checklist
- Layout model + column-aware ordering; engine's recovery/order used and verified
- Per-region processing: text/table/figure/header-footer/key-value
- Region coverage and reading-order checks → retry ladder
- Form schemas per template; value validation; structured + text output
- Table validation; VLM fallback for hard tables
- Template fingerprinting for high-volume forms
- Multi-page table/form continuation at reassembly
- Human-review UI with page image + bboxes; corrections captured
- Layout and table model versions stored per element
Metrics to watch
| Metric | Why |
|---|---|
| Region coverage ratio per page | Missed content |
| Reading-order failure rate | Interleaving |
| Form field validation pass rate per template | Extraction quality |
| Table structure validation pass rate | Table quality |
| $/page vs plain OCR | Routing discipline |
Related
- Per-page routing, Scanned page OCR
- Reading order, Tables
- VLM captioning — fallback for hard tables/figures
- Retry ladder