OCR / digitization → Formula-heavy page (MinerU / LaTeX extraction)

LaTeX Formula Extraction for RAG: MinerU Explained

What it is

Scientific papers, engineering specs, actuarial tables, and financial models are full of mathematics. In a PDF, an equation is a scatter of glyphs from math fonts at fractional positions; the text layer yields symbol soup in the wrong order, and OCR does worse. Neither preserves the structure (fraction, subscript, summation limits) that gives the formula meaning.

Formula-aware parsers detect math regions (inline and display) and run a math recognition model that outputs LaTeX. MinerU (OpenDataLab) is a full pipeline: layout → text → tables → formula detection → formula recognition (UniMERNet) → Markdown with embedded $...$ / $$...$$. Nougat is an end-to-end vision model that transcribes academic pages to Markdown+LaTeX. pix2tex/LaTeX-OCR recognise a single cropped formula image.

Output: Markdown where formulas are LaTeX strings. The LLM reads LaTeX fluently; embeddings capture it reasonably; users can render it.

How to implement

  1. Route pages with math signals (per-page classification and routing): math fonts (CMSY, CMMI, MathJax, STIX), high ratio of Unicode math symbols, \ or $ in extracted text, or a layout model detecting equation regions.
  2. Run MinerU on those pages (it handles layout, text, tables and formulas together; use the same instance for the whole document if the document is math-heavy overall).
  3. Post-process the Markdown: normalise display math to $$ ... $$ on its own lines; keep inline math as $...$; ensure formulas are attached to the preceding text block (an equation is usually the object of the sentence before it).
  4. Chunk formula-aware (layout / structure-aware): a display equation is atomic and keeps with its introducing sentence and its "where x is…" explanation after.
  5. Store both LaTeX and a plain-text rendering (\frac{a}{b} → "a over b" via a simple converter or the LLM) — LaTeX for display and exact search, plain text to help embeddings and BM25 match natural-language questions.
  6. Validate: LaTeX must compile (or at least balance braces/environments); equation count per page should roughly match detected math regions.
  7. Version: MinerU version + formula model weights on every element.

Small example

# MinerU CLI produces markdown + JSON layout; here we post-process the markdown.
import re, subprocess, json, pathlib

def mineru(pdf: str, out: str) -> str:
    subprocess.run(["magic-pdf", "-p", pdf, "-o", out, "-m", "auto"], check=True)
    md = next(pathlib.Path(out).rglob("*.md"))
    return md.read_text()

DISPLAY = re.compile(r"\$\$(.+?)\$\$", re.S)
INLINE = re.compile(r"(?<!\$)\$(?!\$)(.+?)(?<!\$)\$(?!\$)")

def balanced(tex: str) -> bool:
    return tex.count("{") == tex.count("}") and tex.count("\\begin") == tex.count("\\end")

def extract_formulas(md: str) -> list[dict]:
    out = []
    for m in DISPLAY.finditer(md):
        out.append({"kind": "display", "latex": m.group(1).strip(), "valid": balanced(m.group(1))})
    for m in INLINE.finditer(md):
        out.append({"kind": "inline", "latex": m.group(1).strip(), "valid": balanced(m.group(1))})
    return out

def latex_to_words(tex: str) -> str:
    """Tiny, lossy verbalisation so BM25/embeddings can match 'sum of squares'."""
    t = tex
    t = re.sub(r"\\frac\{(.+?)\}\{(.+?)\}", r"(\1) over (\2)", t)
    t = t.replace("\\sum", " sum ").replace("\\int", " integral ").replace("\\sqrt", " square root ")
    t = re.sub(r"[\\{}^_]", " ", t)
    return re.sub(r"\s+", " ", t).strip()

md = mineru("paper.pdf", "/tmp/out")
for f in extract_formulas(md)[:5]:
    print(f["kind"], f["valid"], f["latex"][:50], "→", latex_to_words(f["latex"])[:60])

Pros

  • Formulas survive as structured, renderable, LLM-readable LaTeX instead of symbol soup or nothing.
  • Search works: "Black–Scholes PDE" or "sum of squared residuals" can match the verbalised form.
  • MinerU is a full pipeline: you get layout, tables, and reading order for the page too, in one pass.
  • Better than generic OCR on academic-style tables and references.

Cons

  • GPU-heavy and slower than Docling/OCR; formula recognition is per-region model inference.
  • Recognition errors produce LaTeX that looks right and means something else (a dropped subscript, − vs -). Hard to detect automatically.
  • Embedding models don't understand LaTeX deeply; without verbalisation, semantic search over formulas is weak.
  • Chunking risk: a formula separated from its "where…" definitions is meaningless.
  • Handwritten math, chemistry notation, and unusual fonts remain hard.

When to use

  • Scientific/technical corpora: papers, textbooks, engineering standards, actuarial/financial methodology docs, patents with equations.
  • Pages the router flags as formula-heavy.

When NOT to use

  • General business documents — the occasional x² in a text page doesn't justify the GPU cost; Docling handles simple inline math adequately.
  • When users never ask about the math (they ask about conclusions): route normally and accept symbol loss, but keep the route metadata so you can re-parse later.

Production at scale

Route the page, not the document. A 40-page paper has 6 formula-dense pages. MinerU on all 40 is affordable; on a 5M-page corpus it isn't. Use the router (per-page classification and routing), and for documents where >30% of pages are math, send the whole document (consistent structure beats mixed parsers).

GPU pool. MinerU loads several models (layout, OCR, table, formula detection, formula recognition). Keep long-lived workers; ~10–20 s per page on a mid-range GPU for dense pages. Queue at page or document level with timeouts.

Dual representation. Index text_indexed = text + verbalised formula and store latex for display. For rigorous search over formulas, consider a symbol-level BM25 field (tokenise LaTeX commands) alongside — "\frac \partial" queries actually work.

Formula–context binding. In the chunker, treat sentence_before + display_equation + following "where …" block as one atomic unit. Verify with a golden set of equation questions (e.g. "what does σ represent in equation 3") — this is the failure that shows up in evals.

Validation. Brace balance, environment balance, and optionally a headless KaTeX/MathJax parse; a failing LaTeX string is stored but flagged, and the page goes to the retry ladder (VLM transcription is often a good second rung for formulas).

Version and reproducibility. Formula recognition models improve quickly; each upgrade changes LaTeX strings → chunk text → embeddings. Store formula_model_version; re-parse only formula-class pages on upgrade (change detection config-driven wave filtered by page class).

Cost visibility. $/page for this route is 5–20× the digital route. Report it per source; if a source's math pages are rarely queried, consider lazy parsing (parse on first retrieval of the document's other pages) — an advanced but real cost lever.

Evaluation. Golden set of formula pages with ground-truth LaTeX; metric: formula-level exact match and a normalised edit distance (tokens), plus downstream QA accuracy on equation questions (test-time validation against a golden sample).

Prod checklist

  • Router detects math pages; whole-document routing when math-dense
  • MinerU (or equivalent) on GPU workers, versions pinned and stored
  • Display math normalised; formula kept with its introducing sentence and definitions
  • LaTeX stored for display; verbalised text indexed for search
  • LaTeX validation (balance/parse) → retry ladder on failure
  • Golden formula set with exact-match / edit-distance metrics
  • $/page tracked; lazy-parse option considered for rarely-queried sources

Metrics to watch

MetricWhy
Formula exact-match / edit distance on golden pagesRecognition quality
Invalid-LaTeX rateSilent errors
Equation-question QA accuracyChunking + recognition together
$/page and pages/sec on this routeCost