Generation & grounding → Citations (span-level, validated)
What it is
A citation is a machine-checkable link from a claim to the evidence for it. Three levels of granularity:
| Level | What's cited | Validation possible | Effort |
|---|---|---|---|
| Document | "Source: handbook.pdf" | Existence only | Trivial, near-useless |
| Chunk | [c14] → chunk id | ID exists in the retrieved set | Standard; the practical default |
| Span | Exact quoted text + character offsets within a chunk | Quote is verbatim in the chunk; offsets resolve | Best; enables precise highlighting |
The production contract: the model emits chunk IDs (and optionally verbatim quotes); your code resolves everything else. The model never writes page numbers, URLs, or document titles — those come from your metadata (metadata on every chunk) via the chunk ID. This eliminates fabricated page numbers entirely.
Validation is the part that matters. An ID that isn't in the retrieved set, or a "quote" that doesn't appear in the cited chunk, is a fabrication — and it must be caught before the user sees it.
How to implement
- Put citations in the output schema (structured outputs & tool use):
claims: [{text, citations: [chunk_id], quote?}], not free-form prose with bracket markers you have to parse. - Give passages short, stable IDs in the prompt (
c1…c8mapped to real chunk IDs, or the chunk IDs directly if short). Short IDs reduce transcription errors. - Validate every citation: the ID exists in this request's retrieved set (not just in the corpus). Unknown ID → drop the claim or flag it.
- Validate quotes where used: normalise whitespace and check the quote is a substring of the cited chunk. Near-misses (>90% fuzzy match) can be repaired to the exact span; below that, treat as fabricated.
- Resolve to display metadata server-side: title, page label, section breadcrumb, source URL, effective date, bounding box (page numbers + bounding boxes).
- Enforce coverage: every factual claim must carry at least one citation. Uncited factual claims are the primary hallucination surface — either drop them, flag them in the UI, or fail the response.
- Render with click-through: citation → highlighted region of the source page, behind the same ACL check as retrieval (authentication & authorization).
Small example
import re, difflib
def normalise_ws(s): return " ".join(s.split()).lower()
def validate_citations(response: dict, retrieved: list[dict]) -> dict:
by_id = {c["chunk_id"]: c for c in retrieved}
issues = []
for claim in response["claims"]:
good = []
for cid in claim.get("citations", []):
chunk = by_id.get(cid)
if not chunk:
issues.append({"claim": claim["text"][:80], "cid": cid, "problem": "id_not_retrieved"}); continue
q = claim.get("quote")
if q:
if normalise_ws(q) in normalise_ws(chunk["text"]):
claim["span"] = locate_span(chunk["text"], q) # char offsets for highlighting
else:
ratio = difflib.SequenceMatcher(None, normalise_ws(q), normalise_ws(chunk["text"])).quick_ratio()
if ratio < 0.9:
issues.append({"claim": claim["text"][:80], "cid": cid, "problem": "quote_not_found"}); continue
good.append(cid)
claim["citations"] = good
claim["uncited"] = not good
response["citation_issues"] = issues
response["uncited_claims"] = [c["text"] for c in response["claims"] if c["uncited"]]
return response
def render_citations(response: dict, retrieved: list[dict], caller) -> list[dict]:
"""Model supplies ids only; titles, pages and links come from metadata."""
by_id = {c["chunk_id"]: c for c in retrieved}
out = []
for i, cid in enumerate(dict.fromkeys(c for cl in response["claims"] for c in cl["citations"]), start=1):
c = by_id[cid]
out.append({"marker": f"[{i}]", "chunk_id": cid, "title": c["title"],
"page": c.get("page_label") or c.get("page"), "section": c.get("breadcrumb"),
"effective_date": c.get("effective_date"),
"url": f"/doc/{c['doc_id']}?v={c['version']}&page={c.get('page')}&hl={c.get('bbox')}"})
return out
def enforce_policy(response, min_cited_ratio=1.0):
factual = [c for c in response["claims"] if is_factual(c["text"])]
cited = [c for c in factual if c["citations"]]
if factual and len(cited) / len(factual) < min_cited_ratio:
log_uncited(response["uncited_claims"])
return degrade_or_abstain(response) # policy decision, not silent pass
return response
Pros
- Verifiability — the property that makes enterprise users trust and adopt the system.
- Catches fabrication mechanically: invalid IDs and non-existent quotes are detectable without a model.
- Debuggability: a wrong answer traces to the exact passage that misled the model.
- Auditability for regulated domains (auditability).
- Span offsets enable highlighting, which is consistently the most-praised feature in document QA products.
Cons
- Models cite imprecisely: correct-looking citations attached to the wrong claim, or a citation to a passage that's topically related but doesn't state the claim. ID validity doesn't prove support — that needs faithfulness checking (faithfulness check).
- Quote requirements increase tokens and can cause the model to over-quote instead of synthesising.
- Coverage enforcement is a UX trade-off: dropping uncited claims can leave stubby answers.
- Span offsets break when chunks are re-parsed with a new parser version (pin
versionin citation links, document versioning).
When to use
Every RAG system with a factual answer surface. Chunk-level as the minimum; span-level wherever you can render highlights.
When NOT to use
- Summarisation of a single user-provided document where the whole document is the source (a single provenance line suffices).
- Conversational/chit-chat turns with no factual claims — don't force citations on "you're welcome".
Production at scale
Never let the model write page numbers or URLs. This is the single highest-value rule. IDs from the model, everything else from metadata. Fabricated page numbers are common and uniquely damaging to trust.
Validate against this request's retrieved set, not the whole corpus. A model that cites a real chunk it was never shown is still fabricating, and it can leak the existence of documents the caller can't see.
Invalid-citation rate is a first-class metric (generation evals). Track it per prompt version and per model version; it moves sharply on model upgrades and is an early warning of a prompt regression.
Repair before rejecting. Fuzzy quote matching (>0.9) usually means whitespace or ellipsis differences; repair the span rather than discarding a good claim. Below the threshold, treat as fabricated.
Citation UX matters as much as correctness: inline markers, a source list with titles and dates, hover previews, and click-through highlighting. Users judge grounding by what they can check in one click.
Deduplicate and cap. Ten citations on one sentence is noise; cap at 2–3 per claim, prefer the highest-reranked chunk, and dedupe identical documents.
Parent–child citations (parent–child): cite the child span for precision while showing the parent for context, so the highlight is tight but the reader sees the surrounding clause.
ACL on the citation link. The document-rendering endpoint must independently enforce the caller's permissions — a citation link is a document link (page numbers + bounding boxes, authentication & authorization).
Version-pin citation links so an answer stored in an audit log still resolves to the text that was actually cited, even after the document is updated (document versioning).
Prod checklist
- Citations in the output schema, not parsed from prose
- Model emits chunk IDs (and optional verbatim quotes) only; all display metadata resolved server-side
- Every citation validated against this request's retrieved set
- Quote validation with fuzzy repair threshold; failures flagged as fabricated
- Coverage policy for uncited factual claims (drop / flag / abstain)
- Citations capped and deduped per claim
- Span offsets → highlight rendering; child span cited, parent shown
- Citation links version-pinned and ACL-enforced at the rendering endpoint
- Invalid-citation rate tracked per prompt and model version
Metrics to watch
| Metric | Why |
|---|---|
| Invalid-citation rate (ID not retrieved) | Fabrication |
| Quote-mismatch rate | Fabrication of evidence |
| Uncited-factual-claim rate | Ungrounded content |
| Citation click-through rate | Whether users verify (trust signal) |
| Citation accuracy vs human judgement (sample) | Does the cited passage actually support the claim |
Related
- Grounded prompting — the contract
- Faithfulness — ID validity ≠ support
- Page numbers and bounding boxes — where highlights come from
- Parent–child Parent–child — cite child, show parent
- Auditability