Advanced RAG → Corrective RAG (CRAG): grade, then rewrite or fall back

Corrective RAG (CRAG) Explained: Grade, Rewrite or Fall Back

What it is

Corrective RAG (Yan et al., 2024) inserts a retrieval evaluator between retrieval and generation, and branches on its verdict:

retrieve ──► grade each document: Correct / Ambiguous / Incorrect
   │
   ├── Correct    → refine (keep the relevant strips, drop the noise) → generate
   ├── Ambiguous  → combine: refined internal knowledge + external search → generate
   └── Incorrect  → discard internal results → external search (web / other corpus) → generate
                                                                       └── nothing good → abstain

The "knowledge refinement" step is part of the original method: split retrieved documents into strips, score each strip, and keep only the relevant ones — a denoising pass that removes the irrelevant 80% of a retrieved passage.

The difference from simple thresholding (score thresholding & abstention) is that CRAG acts on a bad verdict — it tries to fix the retrieval — rather than only deciding whether to answer.

How to implement

  1. Choose the grader. Cheapest reliable option first: the reranker score you already compute (reranking). Better for nuance: a small LLM grading each document {relevant: yes|partial|no, reason}. Cheapest at scale: a fine-tuned small classifier (the original paper uses a lightweight T5 evaluator).
  2. Aggregate to a verdict: e.g. any document scoring ≥ τ_high → Correct; all below τ_low → Incorrect; otherwise Ambiguous.
  3. Correct branch — optional strip refinement: split each passage into sentences/strips, score, and keep only relevant strips. This cuts context tokens substantially and reduces distraction (failure modes: poisoning, distraction, confusion, clash).
  4. Ambiguous branch — rewrite the query (a focused, keyword-oriented rewrite works best) and retrieve again, and/or add an external source; merge and re-grade once.
  5. Incorrect branch — fall back to the configured external source (web search, a secondary corpus, a live API), or abstain if no fallback is permitted.
  6. Bound the correction loop: at most one or two correction attempts, with budgets (cost & step budgets).
  7. Log the branch taken on every query — the branch distribution is a direct measure of retrieval health.

Small example

GRADE_SYS = """Grade whether this document contains information useful for answering the question.
Output JSON: {"grade": "yes"|"partial"|"no", "useful_strips": [int], "reason": string}
"useful_strips" lists the indices of sentences that are actually relevant; [] if none."""

def grade_document(question: str, doc: dict) -> dict:
    strips = split_sentences(doc["text"])
    numbered = "\n".join(f"[{i}] {s}" for i, s in enumerate(strips))
    g = json.loads(llm("crag_grade", GRADE_SYS, [{"role": "user",
        "content": f"<question>{question}</question>\n<document>{numbered}</document>"}], max_tokens=200).text)
    g["refined_text"] = " ".join(strips[i] for i in g.get("useful_strips", []) if i < len(strips))
    return g

def corrective_rag(question, caller, allow_web=True):
    docs = rerank_hosted(question, hybrid_retrieve(question, caller), top_n=6)
    grades = [grade_document(question, d) for d in docs]                      # parallelise in production
    yes = [(d, g) for d, g in zip(docs, grades) if g["grade"] == "yes"]
    partial = [(d, g) for d, g in zip(docs, grades) if g["grade"] == "partial"]

    if yes:
        branch = "correct"
        evidence = [d | {"text": g["refined_text"] or d["text"]} for d, g in yes]
    elif partial:
        branch = "ambiguous"
        refined_q = llm("crag_rewrite", REWRITE_KEYWORDS_SYS, [{"role": "user", "content": question}]).text
        more = rerank_hosted(refined_q, hybrid_retrieve(refined_q, caller), top_n=4)
        extra = web_search(refined_q, k=3) if allow_web else []
        evidence = [d | {"text": g["refined_text"] or d["text"]} for d, g in partial] + more + extra
    else:
        branch = "incorrect"
        if not allow_web:
            log_branch(question, branch, abstained=True)
            return abstain(question, [])
        evidence = web_search(llm("crag_rewrite", REWRITE_KEYWORDS_SYS,
                                  [{"role": "user", "content": question}]).text, k=5)

    log_branch(question, branch, n_evidence=len(evidence))
    if not evidence:
        return abstain(question, [])
    return generate(question, evidence, provenance_note=(branch != "correct"))

Pros

  • Prevents generating from irrelevant context — a targeted fix for "confidently wrong on a fresh question" (common failure modes).
  • Strip refinement cuts context noise and tokens, often 50–70% of retrieved text, improving both cost and faithfulness.
  • Graceful degradation path: a bad internal retrieval becomes a web-augmented or honest answer rather than a hallucination.
  • The branch distribution is excellent telemetry — a rising "incorrect" rate is an early warning of corpus or retrieval decay.

Cons

  • Grading costs an LLM call per document (or a model to host) and adds 200–800 ms — parallelise, or use reranker scores as the grader.
  • Grader errors cut both ways: a false "incorrect" throws away good context; a false "correct" defeats the purpose.
  • Web fallback is a governance decision: sending internal questions to a public search engine leaks intent, and web results carry injection risk (prompt-injection defence) and unverifiable provenance.
  • Extra latency on exactly the hard queries users are already waiting longest for.
  • Strip refinement can remove context needed to interpret the kept strips.

When to use

  • Corpora with variable coverage where out-of-scope questions are common.
  • Systems where a web or secondary-corpus fallback is genuinely acceptable.
  • When faithfulness evals show answers generated from weakly relevant context.
  • As a cheaper alternative to full agentic RAG (agentic RAG) when you only need "retry once, then fall back".

When NOT to use

  • Closed-domain systems where external fallback is forbidden (most regulated deployments) — then CRAG reduces to grading + abstention, which is score thresholding & abstention plus a query rewrite; implement that simpler thing.
  • Low-latency paths where per-document grading is unaffordable — use reranker-score thresholds instead.
  • When retrieval is simply weak: fix hybrid/reranking/chunking first. CRAG is a safety net, not a retrieval fix.

Production at scale

Use the reranker as the grader wherever possible. You already compute a calibrated relevance score per passage (reranking). Mapping score bands to Correct/Ambiguous/Incorrect gets most of CRAG's benefit at zero extra latency. Reserve LLM grading for the ambiguous band only.

Parallelise grading across documents with a bounded concurrency, and cap it (grade the top 6, not the top 50).

Web fallback governance. Explicit per-tenant setting; queries sanitised (never send raw internal text or identifiers); results clearly labelled as external in the answer and in citations (citations); domain allow-lists for regulated tenants; and treat web content as untrusted data with the full injection defences (prompt-injection defence, delimiters: instructions vs data).

Strip refinement pays for itself in token cost when passages are long (parents, parent–child). Measure tokens saved vs grading cost; on long-context pipelines it's usually net positive.

Branch distribution as an SLI. Track correct / ambiguous / incorrect rates per tenant and per query type, weekly. A shift toward "incorrect" means corpus drift, an ingestion failure, or a new question type — often before users complain.

Bound corrections to one round. A second rewrite-and-retry rarely helps and doubles the worst-case latency. If one correction doesn't produce usable evidence, abstain.

Provenance in the answer. When evidence came from the fallback path, say so: "I couldn't find this in internal documents; based on public sources…". This is a correctness and a trust requirement.

Evaluate the grader itself against human labels on a sample (LLM-as-judge) — grader precision/recall is the component that determines whether CRAG helps or hurts.

Prod checklist

  • Reranker score as the primary grader; LLM grading only for the ambiguous band
  • Grading parallelised and capped to top-N documents
  • Strip refinement with a fallback to full passage when no strips are kept
  • Exactly one correction round; then answer or abstain
  • Web/secondary fallback gated per tenant; queries sanitised; results labelled external and treated as untrusted
  • Branch (correct/ambiguous/incorrect) logged per query; weekly distribution dashboard
  • Answer states when evidence came from the fallback path
  • Grader accuracy measured against human labels on a sample

Metrics to watch

MetricWhy
Branch distribution over time and per tenantRetrieval/corpus health
Grader precision/recall vs human labelsWhether the safety net works
Tokens saved by strip refinement vs grading costNet economics
Faithfulness on ambiguous/incorrect branchesFallback answer quality