Generation & grounding → Citations (span-level, validated)

RAG Citations Explained: Validate Chunk IDs and Quotes

What it is

A citation is a machine-checkable link from a claim to the evidence for it. Three levels of granularity:

LevelWhat's citedValidation possibleEffort
Document"Source: handbook.pdf"Existence onlyTrivial, near-useless
Chunk[c14] → chunk idID exists in the retrieved setStandard; the practical default
SpanExact quoted text + character offsets within a chunkQuote is verbatim in the chunk; offsets resolveBest; enables precise highlighting

The production contract: the model emits chunk IDs (and optionally verbatim quotes); your code resolves everything else. The model never writes page numbers, URLs, or document titles — those come from your metadata (metadata on every chunk) via the chunk ID. This eliminates fabricated page numbers entirely.

Validation is the part that matters. An ID that isn't in the retrieved set, or a "quote" that doesn't appear in the cited chunk, is a fabrication — and it must be caught before the user sees it.

How to implement

  1. Put citations in the output schema (structured outputs & tool use): claims: [{text, citations: [chunk_id], quote?}], not free-form prose with bracket markers you have to parse.
  2. Give passages short, stable IDs in the prompt (c1…c8 mapped to real chunk IDs, or the chunk IDs directly if short). Short IDs reduce transcription errors.
  3. Validate every citation: the ID exists in this request's retrieved set (not just in the corpus). Unknown ID → drop the claim or flag it.
  4. Validate quotes where used: normalise whitespace and check the quote is a substring of the cited chunk. Near-misses (>90% fuzzy match) can be repaired to the exact span; below that, treat as fabricated.
  5. Resolve to display metadata server-side: title, page label, section breadcrumb, source URL, effective date, bounding box (page numbers + bounding boxes).
  6. Enforce coverage: every factual claim must carry at least one citation. Uncited factual claims are the primary hallucination surface — either drop them, flag them in the UI, or fail the response.
  7. Render with click-through: citation → highlighted region of the source page, behind the same ACL check as retrieval (authentication & authorization).

Small example

import re, difflib

def normalise_ws(s): return " ".join(s.split()).lower()

def validate_citations(response: dict, retrieved: list[dict]) -> dict:
    by_id = {c["chunk_id"]: c for c in retrieved}
    issues = []
    for claim in response["claims"]:
        good = []
        for cid in claim.get("citations", []):
            chunk = by_id.get(cid)
            if not chunk:
                issues.append({"claim": claim["text"][:80], "cid": cid, "problem": "id_not_retrieved"}); continue
            q = claim.get("quote")
            if q:
                if normalise_ws(q) in normalise_ws(chunk["text"]):
                    claim["span"] = locate_span(chunk["text"], q)                 # char offsets for highlighting
                else:
                    ratio = difflib.SequenceMatcher(None, normalise_ws(q), normalise_ws(chunk["text"])).quick_ratio()
                    if ratio < 0.9:
                        issues.append({"claim": claim["text"][:80], "cid": cid, "problem": "quote_not_found"}); continue
            good.append(cid)
        claim["citations"] = good
        claim["uncited"] = not good
    response["citation_issues"] = issues
    response["uncited_claims"] = [c["text"] for c in response["claims"] if c["uncited"]]
    return response

def render_citations(response: dict, retrieved: list[dict], caller) -> list[dict]:
    """Model supplies ids only; titles, pages and links come from metadata."""
    by_id = {c["chunk_id"]: c for c in retrieved}
    out = []
    for i, cid in enumerate(dict.fromkeys(c for cl in response["claims"] for c in cl["citations"]), start=1):
        c = by_id[cid]
        out.append({"marker": f"[{i}]", "chunk_id": cid, "title": c["title"],
                    "page": c.get("page_label") or c.get("page"), "section": c.get("breadcrumb"),
                    "effective_date": c.get("effective_date"),
                    "url": f"/doc/{c['doc_id']}?v={c['version']}&page={c.get('page')}&hl={c.get('bbox')}"})
    return out

def enforce_policy(response, min_cited_ratio=1.0):
    factual = [c for c in response["claims"] if is_factual(c["text"])]
    cited = [c for c in factual if c["citations"]]
    if factual and len(cited) / len(factual) < min_cited_ratio:
        log_uncited(response["uncited_claims"])
        return degrade_or_abstain(response)                                       # policy decision, not silent pass
    return response

Pros

  • Verifiability — the property that makes enterprise users trust and adopt the system.
  • Catches fabrication mechanically: invalid IDs and non-existent quotes are detectable without a model.
  • Debuggability: a wrong answer traces to the exact passage that misled the model.
  • Auditability for regulated domains (auditability).
  • Span offsets enable highlighting, which is consistently the most-praised feature in document QA products.

Cons

  • Models cite imprecisely: correct-looking citations attached to the wrong claim, or a citation to a passage that's topically related but doesn't state the claim. ID validity doesn't prove support — that needs faithfulness checking (faithfulness check).
  • Quote requirements increase tokens and can cause the model to over-quote instead of synthesising.
  • Coverage enforcement is a UX trade-off: dropping uncited claims can leave stubby answers.
  • Span offsets break when chunks are re-parsed with a new parser version (pin version in citation links, document versioning).

When to use

Every RAG system with a factual answer surface. Chunk-level as the minimum; span-level wherever you can render highlights.

When NOT to use

  • Summarisation of a single user-provided document where the whole document is the source (a single provenance line suffices).
  • Conversational/chit-chat turns with no factual claims — don't force citations on "you're welcome".

Production at scale

Never let the model write page numbers or URLs. This is the single highest-value rule. IDs from the model, everything else from metadata. Fabricated page numbers are common and uniquely damaging to trust.

Validate against this request's retrieved set, not the whole corpus. A model that cites a real chunk it was never shown is still fabricating, and it can leak the existence of documents the caller can't see.

Invalid-citation rate is a first-class metric (generation evals). Track it per prompt version and per model version; it moves sharply on model upgrades and is an early warning of a prompt regression.

Repair before rejecting. Fuzzy quote matching (>0.9) usually means whitespace or ellipsis differences; repair the span rather than discarding a good claim. Below the threshold, treat as fabricated.

Citation UX matters as much as correctness: inline markers, a source list with titles and dates, hover previews, and click-through highlighting. Users judge grounding by what they can check in one click.

Deduplicate and cap. Ten citations on one sentence is noise; cap at 2–3 per claim, prefer the highest-reranked chunk, and dedupe identical documents.

Parent–child citations (parent–child): cite the child span for precision while showing the parent for context, so the highlight is tight but the reader sees the surrounding clause.

ACL on the citation link. The document-rendering endpoint must independently enforce the caller's permissions — a citation link is a document link (page numbers + bounding boxes, authentication & authorization).

Version-pin citation links so an answer stored in an audit log still resolves to the text that was actually cited, even after the document is updated (document versioning).

Prod checklist

  • Citations in the output schema, not parsed from prose
  • Model emits chunk IDs (and optional verbatim quotes) only; all display metadata resolved server-side
  • Every citation validated against this request's retrieved set
  • Quote validation with fuzzy repair threshold; failures flagged as fabricated
  • Coverage policy for uncited factual claims (drop / flag / abstain)
  • Citations capped and deduped per claim
  • Span offsets → highlight rendering; child span cited, parent shown
  • Citation links version-pinned and ACL-enforced at the rendering endpoint
  • Invalid-citation rate tracked per prompt and model version

Metrics to watch

MetricWhy
Invalid-citation rate (ID not retrieved)Fabrication
Quote-mismatch rateFabrication of evidence
Uncited-factual-claim rateUngrounded content
Citation click-through rateWhether users verify (trust signal)
Citation accuracy vs human judgement (sample)Does the cited passage actually support the claim