OCR / digitization → Charts, images, diagrams (VLM captioning)
What it is
Figures carry facts: the bar chart with Q3 revenue by region, the architecture diagram, the flowchart of the approval process, the photo of the damaged part. Text parsers return the caption at best. OCR returns the axis labels as random words. The knowledge in the picture is invisible to retrieval.
VLM captioning sends the figure image (plus surrounding context: caption, section title, nearby text) to a vision-language model (Claude, GPT-4o-class, Qwen-VL, LLaVA, PaliGemma) and asks for a dense, factual description: what kind of figure, what it shows, the values, trends, labels, relationships. That text is indexed as an element — a chunk of its own with type=figure — with a reference to the stored image.
Two flavours:
- Description for retrieval (this file): text in the index.
- Image retrieval (multimodal RAG multimodal RAG, ColPali-style): embed page images directly. Complementary; captioning is cheaper and works with a text-only pipeline.
How to implement
- Detect figure regions (layout model, complex layout / Docling
PictureItem) and crop them at good resolution; skip tiny images (icons, logos) by size/aspect thresholds. - Gather context: caption text, section breadcrumb, 1–2 sentences before/after the figure reference ("as shown in Figure 3, …").
- Prompt the VLM with the image + context, asking for: figure type; a one-sentence summary; the detailed content (for charts: axes, series, values or trends, notable points; for diagrams: components and relationships; for tables-as-images: transcribe to Markdown; for screenshots: UI elements and text); keywords. Temperature 0. Ask for a fixed JSON schema.
- Validate: non-empty, length bounds, JSON parses, no refusal text, language matches document.
- Emit an element
{type: "figure", text: description, caption, image_ref, bbox, page, vlm_model_version}; the chunker keeps it as an atomic chunk with the breadcrumb prepended. - Store the image (object storage, keyed by
doc_id/version/page/figure_idx) so the answer path can attach it to the LLM call when the figure is retrieved (multimodal generation). - Cache descriptions by
sha256(image bytes + prompt version).
Small example
import anthropic, base64, hashlib, json
client = anthropic.Anthropic()
PROMPT_V = "fig-desc-v3"
SCHEMA = {"figure_type": "chart|diagram|table_image|photo|screenshot|other",
"summary": "one sentence", "details": "dense factual description; for charts list series and values/trends",
"keywords": ["..."]}
def describe_figure(png_bytes: bytes, caption: str, breadcrumb: str, nearby: str, cache: dict) -> dict:
key = hashlib.sha256(png_bytes + PROMPT_V.encode()).hexdigest()
if key in cache:
return cache[key]
resp = client.messages.create(
model="claude-sonnet-5", max_tokens=600, temperature=0,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png",
"data": base64.b64encode(png_bytes).decode()}},
{"type": "text", "text":
f"Document section: {breadcrumb}\nCaption: {caption}\nNearby text: {nearby}\n\n"
f"Describe this figure for search indexing. Be factual and specific; include numbers you can read. "
f"Return JSON exactly matching: {json.dumps(SCHEMA)}"},
]}],
)
txt = resp.content[0].text
data = json.loads(txt[txt.find("{"): txt.rfind("}") + 1])
assert data.get("details") and len(data["details"]) < 4000
cache[key] = data
return data
# element text for indexing:
# f"[{breadcrumb}] Figure ({d['figure_type']}): {caption}. {d['summary']} {d['details']} Keywords: {', '.join(d['keywords'])}"
Pros
- Makes visual knowledge retrievable with a text-only RAG stack.
- Charts become numbers: "Q3 EMEA revenue was ~$4.2M, up from $3.8M" is now a searchable sentence.
- Works for diagrams and screenshots that OCR mangles.
- Stored image + retrieved description lets the answer step re-look at the image for precision.
Cons
- Cost: a VLM call per figure; large corpora have millions of figures.
- Hallucinated values: VLMs misread charts (estimating bar heights, confusing series). A confident wrong number in the index is worse than no number.
- Description quality depends on context: without the caption and section, the model describes a "blue bar chart".
- Latency at ingest; figure-heavy documents parse slowly.
- Decorative images waste calls unless filtered.
- Data governance: images go to a model provider unless self-hosted.
When to use
- Corpora where figures carry facts: financial reports, slide decks, engineering docs, product manuals, medical imaging reports (with care), UI documentation.
- When users ask "what did the chart show" and get nothing.
- As the fallback rung for tables and formulas the specialised models fail on (bounded escalation with visible failure).
When NOT to use
- Corpora where images are decorative (marketing photos, stock images) — filter by size/aspect and skip.
- Precision-critical numeric extraction from charts — a VLM estimate is not a source of truth; extract from the underlying data table or tag the description as approximate.
- Highly sensitive images without an approved (self-hosted or zero-retention) VLM path.
Production at scale
Filter before you pay. Skip images < ~150 px on a side, extreme aspect ratios (rules, dividers), and images that recur across many documents (logos; detect by perceptual hash frequency). Typically removes 50–80% of image regions.
Tiering. Cheap small VLM (self-hosted Qwen-VL / PaliGemma, or a Haiku-class model) for a first description; escalate to a stronger model only for figures with numbers/tables or when the cheap description fails validation. Batch API for backfills.
Mark uncertainty explicitly. Prompt for values_confidence: exact|approximate|unreadable and prefix approximate numbers with "approximately" in the indexed text. The generator then hedges correctly, and the faithfulness checker (faithfulness check) doesn't flag it as hallucination.
Attach the image at answer time. When a figure chunk is retrieved, include the stored image in the generation call (multimodal). The description gets the chunk retrieved; the image makes the answer accurate. Budget for it (per-source truncation budgets) — images cost tokens.
Caching and versioning. Cache by image hash + prompt version; store vlm_model_version and prompt_version on the element. A prompt or model change is a re-caption wave for figure elements only (filter by element type).
Deduplicate figures. The same chart appears in the deck, the report, and the email. Perceptual-hash dedupe (pHash) within tenant; caption once, reference many.
Quality gate. Sample 1–2% of captions to human review; measure "description is factually consistent with image" and "numbers correct" (test-time validation against a golden sample). Track hallucinated-value rate per model version; it's the number that decides whether to trust chart values at all.
Governance. Confirm zero-retention and region for the VLM (data residency & retention). For regulated tenants, self-hosted VLM only; make the model choice a per-tenant setting.
Multimodal retrieval alternative. If figure-heavy and budget allows, add page-image embeddings (multimodal RAG) — retrieval by visual similarity catches figures whose description missed the user's angle. Captioning remains the cheaper baseline.
Prod checklist
- Figure detection + size/aspect/logo filters
- Context (caption, breadcrumb, nearby text) passed to the VLM
- Fixed JSON schema; validation; uncertainty tag for numbers
- Tiered models; batch API for backfills; cache by image hash + prompt version
- Image stored and attached at generation time when the figure chunk is retrieved
- pHash dedupe of figures within tenant
- Human-review sample; hallucinated-value rate tracked per model version
- Zero-retention / self-hosted VLM per tenant policy
-
vlm_model_version+prompt_versionon elements; re-caption wave procedure
Metrics to watch
| Metric | Why |
|---|---|
| Figures captioned / detected (post-filter ratio) | Cost discipline |
| $ per figure; cache hit rate | Cost |
| Hallucinated-value rate (human sample) | Trust in chart numbers |
| Figure-question answer accuracy on golden set | The reason for doing this |
| Validation failure rate per model version | Regression |
Related
- Per-page routing, Complex layout (figure regions)
- Multimodal RAG — image embeddings as a complement
- Retry ladder — VLM as a rung for tables/formulas
- Faithfulness check — handle "approximately" correctly
- Data residency