Structure extraction → Reading order

PDF Reading Order in RAG Parsing Explained: Why Text Gets Scrambled

What it is

A PDF stores glyphs at coordinates; a scanned page stores pixels. Neither stores "read this paragraph, then that one". Parsers must infer reading order. Failure modes:

  • Column interleaving: two-column text emitted line-by-line across columns.
  • Sidebars/callouts injected mid-paragraph.
  • Footnotes appearing where the marker is, or between paragraphs.
  • Table cells interleaved with body text.
  • Rotated text, headers/footers in the middle of content.
  • Right-to-left scripts and vertical CJK.

Good parsers (Docling, MinerU, PP-Structure with recovery) use a layout model plus a reading-order model or XY-cut heuristic; still, verification is your job because errors are silent — the text is all there, just scrambled.

How to implement

  1. Use a layout-aware parser and its ordering output rather than raw extraction (digital document, complex layout).
  2. Column-aware fallback: cluster text blocks by x-position into columns; sort by column then y; then merge. XY-cut (recursively split the page at the widest horizontal/vertical gap) handles most layouts.
  3. Separate floating elements: tables, figures, captions, footnotes, sidebars, headers/footers are emitted as their own elements anchored to a position, not spliced into paragraphs.
  4. Sentence-continuity check: for consecutive text elements, if element A ends without terminal punctuation and element B starts lowercase (or vice versa in a suspicious way), that boundary is a candidate error. Score the page; high score → escalate (bounded escalation with visible failure).
  5. Hyphenation and line joins: join lines within a paragraph, de-hyphenate line-end hyphens (inter-\nnational → international) using a dictionary check (keep genuine hyphens).
  6. Language direction: detect RTL/vertical scripts and apply the correct ordering rule.
  7. Store reading_index on each element so the order is explicit and auditable.

Small example

def xy_cut(blocks, gap=18):
    """blocks: [(x0,y0,x1,y1,text)]. Recursive split on the widest gap; returns ordered blocks."""
    if len(blocks) <= 1:
        return blocks
    for axis in (0, 1):                                    # try vertical cut (columns) first, then horizontal
        lo, hi = (0, 2) if axis == 0 else (1, 3)
        s = sorted(blocks, key=lambda b: b[lo])
        best, best_gap = None, gap
        for i in range(1, len(s)):
            g = s[i][lo] - max(b[hi] for b in s[:i])
            if g > best_gap:
                best, best_gap = i, g
        if best:
            return xy_cut(s[:best], gap) + xy_cut(s[best:], gap)
    return sorted(blocks, key=lambda b: (b[1], b[0]))       # no gap: top-to-bottom, left-to-right

def continuity_score(ordered_texts):
    bad = 0
    for a, b in zip(ordered_texts, ordered_texts[1:]):
        a, b = a.rstrip(), b.lstrip()
        if a and b and a[-1] not in ".!?:;\"')" and b[0].isupper():
            bad += 1                                        # A trails off mid-sentence, B starts a new one
    return bad / max(len(ordered_texts) - 1, 1)

import re
def dehyphenate(lines, vocab):
    out = []
    for i, ln in enumerate(lines):
        if ln.endswith("-") and i + 1 < len(lines):
            head, nxt = ln[:-1].split()[-1], lines[i + 1].split()[0]
            if (head + nxt).lower() in vocab:
                lines[i + 1] = lines[i + 1].replace(nxt, head + nxt, 1); ln = ln[: -len(head) - 1]
        out.append(ln)
    return " ".join(out)

Pros

  • Sentences stay whole → embeddings mean something, and the LLM reads coherent text.
  • Chunk boundaries fall on real paragraphs (recursive character splitting/04.1.3 depend on this).
  • Tables and footnotes stop polluting body paragraphs.
  • Explicit reading_index makes citations and highlighting consistent with what the user reads.

Cons

  • Heuristics fail on creative layouts (magazines, marketing, infographics).
  • Layout model errors cascade: a missed column boundary scrambles the page.
  • Verification is approximate: continuity checks miss errors on pages of short lines (lists, addresses).
  • Extra processing per page; small but real.

When to use

Always — verification is cheap, and errors are silent. Especially for multi-column sources (papers, newsletters, statements, regulatory filings).

When NOT to use (the heavy machinery)

  • Single-column born-digital documents and Office formats where order is explicit — just trust it, but keep the continuity check as a smoke test.

Production at scale

Trust the parser, verify the page. Compute the continuity score per page in the validation stage (production per-page heuristics). Threshold it per source (learn from the golden set). Failing pages → retry ladder with an alternative ordering strategy (XY-cut vs the parser's learned order) before escalating to a VLM.

Floating elements as first-class elements. The single most effective design decision: never splice a table, figure, footnote, or sidebar into paragraph text. Emit them as separate elements with an anchor_reading_index so the chunker can keep a table near its referencing paragraph without breaking the paragraph.

Footnotes. Detect by position (bottom region), small font, and leading marker; emit as footnote elements linked to their marker; append to the chunk of the referencing paragraph or index separately with the breadcrumb. Legal and financial corpora put material facts in footnotes.

Two-page spreads and rotated pages. Detect page rotation (PyMuPDF page.rotation, or from OCR angle classification) and landscape spreads; split spreads into two logical pages before ordering.

Language and script. RTL (Arabic, Hebrew) and vertical CJK need script-aware ordering; detect per page and switch rules. Mixed-script pages (Arabic with Latin numbers) are the hardest; test them explicitly.

Golden set for order. Keep ~50 pages with human-verified order; metric: pairwise order accuracy or Kendall tau over blocks (test-time validation against a golden sample). Run on parser upgrades.

Downstream tells. If chunk-level recall is poor on a multi-column source while single-column sources are fine, suspect ordering before anything else. Keep page_class and source in eval breakdowns.

Throughput. XY-cut and continuity checks are microseconds per page; de-hyphenation with a vocabulary lookup is cheap. No capacity concern; the concern is only doing it.

Prod checklist

  • Layout-aware parser ordering used; XY-cut fallback available
  • Tables/figures/footnotes/sidebars/headers as separate anchored elements
  • Per-page continuity score in validation; per-source threshold
  • De-hyphenation with dictionary check
  • Rotation/spread handling; script-aware ordering for RTL/vertical
  • reading_index stored on elements
  • Golden pages with verified order; order metric in the parsing eval
  • Eval breakdown by page class/source to spot ordering regressions

Metrics to watch

MetricWhy
Continuity score distribution per sourceOrdering errors
Order accuracy (Kendall tau) on golden pagesParser upgrades
Retry-ladder escalations for orderingParser fit per source
Chunk recall by page classDownstream effect