OCR / digitization → Formula-heavy page (MinerU / LaTeX extraction)
What it is
Scientific papers, engineering specs, actuarial tables, and financial models are full of mathematics. In a PDF, an equation is a scatter of glyphs from math fonts at fractional positions; the text layer yields symbol soup in the wrong order, and OCR does worse. Neither preserves the structure (fraction, subscript, summation limits) that gives the formula meaning.
Formula-aware parsers detect math regions (inline and display) and run a math recognition model that outputs LaTeX. MinerU (OpenDataLab) is a full pipeline: layout → text → tables → formula detection → formula recognition (UniMERNet) → Markdown with embedded $...$ / $$...$$. Nougat is an end-to-end vision model that transcribes academic pages to Markdown+LaTeX. pix2tex/LaTeX-OCR recognise a single cropped formula image.
Output: Markdown where formulas are LaTeX strings. The LLM reads LaTeX fluently; embeddings capture it reasonably; users can render it.
How to implement
- Route pages with math signals (per-page classification and routing): math fonts (
CMSY,CMMI,MathJax,STIX), high ratio of Unicode math symbols,\or$in extracted text, or a layout model detectingequationregions. - Run MinerU on those pages (it handles layout, text, tables and formulas together; use the same instance for the whole document if the document is math-heavy overall).
- Post-process the Markdown: normalise display math to
$$ ... $$on its own lines; keep inline math as$...$; ensure formulas are attached to the preceding text block (an equation is usually the object of the sentence before it). - Chunk formula-aware (layout / structure-aware): a display equation is atomic and keeps with its introducing sentence and its "where x is…" explanation after.
- Store both LaTeX and a plain-text rendering (
\frac{a}{b}→ "a over b" via a simple converter or the LLM) — LaTeX for display and exact search, plain text to help embeddings and BM25 match natural-language questions. - Validate: LaTeX must compile (or at least balance braces/environments); equation count per page should roughly match detected math regions.
- Version: MinerU version + formula model weights on every element.
Small example
# MinerU CLI produces markdown + JSON layout; here we post-process the markdown.
import re, subprocess, json, pathlib
def mineru(pdf: str, out: str) -> str:
subprocess.run(["magic-pdf", "-p", pdf, "-o", out, "-m", "auto"], check=True)
md = next(pathlib.Path(out).rglob("*.md"))
return md.read_text()
DISPLAY = re.compile(r"\$\$(.+?)\$\$", re.S)
INLINE = re.compile(r"(?<!\$)\$(?!\$)(.+?)(?<!\$)\$(?!\$)")
def balanced(tex: str) -> bool:
return tex.count("{") == tex.count("}") and tex.count("\\begin") == tex.count("\\end")
def extract_formulas(md: str) -> list[dict]:
out = []
for m in DISPLAY.finditer(md):
out.append({"kind": "display", "latex": m.group(1).strip(), "valid": balanced(m.group(1))})
for m in INLINE.finditer(md):
out.append({"kind": "inline", "latex": m.group(1).strip(), "valid": balanced(m.group(1))})
return out
def latex_to_words(tex: str) -> str:
"""Tiny, lossy verbalisation so BM25/embeddings can match 'sum of squares'."""
t = tex
t = re.sub(r"\\frac\{(.+?)\}\{(.+?)\}", r"(\1) over (\2)", t)
t = t.replace("\\sum", " sum ").replace("\\int", " integral ").replace("\\sqrt", " square root ")
t = re.sub(r"[\\{}^_]", " ", t)
return re.sub(r"\s+", " ", t).strip()
md = mineru("paper.pdf", "/tmp/out")
for f in extract_formulas(md)[:5]:
print(f["kind"], f["valid"], f["latex"][:50], "→", latex_to_words(f["latex"])[:60])
Pros
- Formulas survive as structured, renderable, LLM-readable LaTeX instead of symbol soup or nothing.
- Search works: "Black–Scholes PDE" or "sum of squared residuals" can match the verbalised form.
- MinerU is a full pipeline: you get layout, tables, and reading order for the page too, in one pass.
- Better than generic OCR on academic-style tables and references.
Cons
- GPU-heavy and slower than Docling/OCR; formula recognition is per-region model inference.
- Recognition errors produce LaTeX that looks right and means something else (a dropped subscript,
−vs-). Hard to detect automatically. - Embedding models don't understand LaTeX deeply; without verbalisation, semantic search over formulas is weak.
- Chunking risk: a formula separated from its "where…" definitions is meaningless.
- Handwritten math, chemistry notation, and unusual fonts remain hard.
When to use
- Scientific/technical corpora: papers, textbooks, engineering standards, actuarial/financial methodology docs, patents with equations.
- Pages the router flags as formula-heavy.
When NOT to use
- General business documents — the occasional
x²in a text page doesn't justify the GPU cost; Docling handles simple inline math adequately. - When users never ask about the math (they ask about conclusions): route normally and accept symbol loss, but keep the route metadata so you can re-parse later.
Production at scale
Route the page, not the document. A 40-page paper has 6 formula-dense pages. MinerU on all 40 is affordable; on a 5M-page corpus it isn't. Use the router (per-page classification and routing), and for documents where >30% of pages are math, send the whole document (consistent structure beats mixed parsers).
GPU pool. MinerU loads several models (layout, OCR, table, formula detection, formula recognition). Keep long-lived workers; ~10–20 s per page on a mid-range GPU for dense pages. Queue at page or document level with timeouts.
Dual representation. Index text_indexed = text + verbalised formula and store latex for display. For rigorous search over formulas, consider a symbol-level BM25 field (tokenise LaTeX commands) alongside — "\frac \partial" queries actually work.
Formula–context binding. In the chunker, treat sentence_before + display_equation + following "where …" block as one atomic unit. Verify with a golden set of equation questions (e.g. "what does σ represent in equation 3") — this is the failure that shows up in evals.
Validation. Brace balance, environment balance, and optionally a headless KaTeX/MathJax parse; a failing LaTeX string is stored but flagged, and the page goes to the retry ladder (VLM transcription is often a good second rung for formulas).
Version and reproducibility. Formula recognition models improve quickly; each upgrade changes LaTeX strings → chunk text → embeddings. Store formula_model_version; re-parse only formula-class pages on upgrade (change detection config-driven wave filtered by page class).
Cost visibility. $/page for this route is 5–20× the digital route. Report it per source; if a source's math pages are rarely queried, consider lazy parsing (parse on first retrieval of the document's other pages) — an advanced but real cost lever.
Evaluation. Golden set of formula pages with ground-truth LaTeX; metric: formula-level exact match and a normalised edit distance (tokens), plus downstream QA accuracy on equation questions (test-time validation against a golden sample).
Prod checklist
- Router detects math pages; whole-document routing when math-dense
- MinerU (or equivalent) on GPU workers, versions pinned and stored
- Display math normalised; formula kept with its introducing sentence and definitions
- LaTeX stored for display; verbalised text indexed for search
- LaTeX validation (balance/parse) → retry ladder on failure
- Golden formula set with exact-match / edit-distance metrics
- $/page tracked; lazy-parse option considered for rarely-queried sources
Metrics to watch
| Metric | Why |
|---|---|
| Formula exact-match / edit distance on golden pages | Recognition quality |
| Invalid-LaTeX rate | Silent errors |
| Equation-question QA accuracy | Chunking + recognition together |
| $/page and pages/sec on this route | Cost |
Related
- Per-page routing
- VLM captioning — fallback rung for hard formulas
- Layout-aware chunking — atomic equation units
- Validation against golden sample