Advanced RAG → Corrective RAG (CRAG): grade, then rewrite or fall back
What it is
Corrective RAG (Yan et al., 2024) inserts a retrieval evaluator between retrieval and generation, and branches on its verdict:
retrieve ──► grade each document: Correct / Ambiguous / Incorrect
│
├── Correct → refine (keep the relevant strips, drop the noise) → generate
├── Ambiguous → combine: refined internal knowledge + external search → generate
└── Incorrect → discard internal results → external search (web / other corpus) → generate
└── nothing good → abstain
The "knowledge refinement" step is part of the original method: split retrieved documents into strips, score each strip, and keep only the relevant ones — a denoising pass that removes the irrelevant 80% of a retrieved passage.
The difference from simple thresholding (score thresholding & abstention) is that CRAG acts on a bad verdict — it tries to fix the retrieval — rather than only deciding whether to answer.
How to implement
- Choose the grader. Cheapest reliable option first: the reranker score you already compute (reranking). Better for nuance: a small LLM grading each document
{relevant: yes|partial|no, reason}. Cheapest at scale: a fine-tuned small classifier (the original paper uses a lightweight T5 evaluator). - Aggregate to a verdict: e.g. any document scoring ≥ τ_high → Correct; all below τ_low → Incorrect; otherwise Ambiguous.
- Correct branch — optional strip refinement: split each passage into sentences/strips, score, and keep only relevant strips. This cuts context tokens substantially and reduces distraction (failure modes: poisoning, distraction, confusion, clash).
- Ambiguous branch — rewrite the query (a focused, keyword-oriented rewrite works best) and retrieve again, and/or add an external source; merge and re-grade once.
- Incorrect branch — fall back to the configured external source (web search, a secondary corpus, a live API), or abstain if no fallback is permitted.
- Bound the correction loop: at most one or two correction attempts, with budgets (cost & step budgets).
- Log the branch taken on every query — the branch distribution is a direct measure of retrieval health.
Small example
GRADE_SYS = """Grade whether this document contains information useful for answering the question.
Output JSON: {"grade": "yes"|"partial"|"no", "useful_strips": [int], "reason": string}
"useful_strips" lists the indices of sentences that are actually relevant; [] if none."""
def grade_document(question: str, doc: dict) -> dict:
strips = split_sentences(doc["text"])
numbered = "\n".join(f"[{i}] {s}" for i, s in enumerate(strips))
g = json.loads(llm("crag_grade", GRADE_SYS, [{"role": "user",
"content": f"<question>{question}</question>\n<document>{numbered}</document>"}], max_tokens=200).text)
g["refined_text"] = " ".join(strips[i] for i in g.get("useful_strips", []) if i < len(strips))
return g
def corrective_rag(question, caller, allow_web=True):
docs = rerank_hosted(question, hybrid_retrieve(question, caller), top_n=6)
grades = [grade_document(question, d) for d in docs] # parallelise in production
yes = [(d, g) for d, g in zip(docs, grades) if g["grade"] == "yes"]
partial = [(d, g) for d, g in zip(docs, grades) if g["grade"] == "partial"]
if yes:
branch = "correct"
evidence = [d | {"text": g["refined_text"] or d["text"]} for d, g in yes]
elif partial:
branch = "ambiguous"
refined_q = llm("crag_rewrite", REWRITE_KEYWORDS_SYS, [{"role": "user", "content": question}]).text
more = rerank_hosted(refined_q, hybrid_retrieve(refined_q, caller), top_n=4)
extra = web_search(refined_q, k=3) if allow_web else []
evidence = [d | {"text": g["refined_text"] or d["text"]} for d, g in partial] + more + extra
else:
branch = "incorrect"
if not allow_web:
log_branch(question, branch, abstained=True)
return abstain(question, [])
evidence = web_search(llm("crag_rewrite", REWRITE_KEYWORDS_SYS,
[{"role": "user", "content": question}]).text, k=5)
log_branch(question, branch, n_evidence=len(evidence))
if not evidence:
return abstain(question, [])
return generate(question, evidence, provenance_note=(branch != "correct"))
Pros
- Prevents generating from irrelevant context — a targeted fix for "confidently wrong on a fresh question" (common failure modes).
- Strip refinement cuts context noise and tokens, often 50–70% of retrieved text, improving both cost and faithfulness.
- Graceful degradation path: a bad internal retrieval becomes a web-augmented or honest answer rather than a hallucination.
- The branch distribution is excellent telemetry — a rising "incorrect" rate is an early warning of corpus or retrieval decay.
Cons
- Grading costs an LLM call per document (or a model to host) and adds 200–800 ms — parallelise, or use reranker scores as the grader.
- Grader errors cut both ways: a false "incorrect" throws away good context; a false "correct" defeats the purpose.
- Web fallback is a governance decision: sending internal questions to a public search engine leaks intent, and web results carry injection risk (prompt-injection defence) and unverifiable provenance.
- Extra latency on exactly the hard queries users are already waiting longest for.
- Strip refinement can remove context needed to interpret the kept strips.
When to use
- Corpora with variable coverage where out-of-scope questions are common.
- Systems where a web or secondary-corpus fallback is genuinely acceptable.
- When faithfulness evals show answers generated from weakly relevant context.
- As a cheaper alternative to full agentic RAG (agentic RAG) when you only need "retry once, then fall back".
When NOT to use
- Closed-domain systems where external fallback is forbidden (most regulated deployments) — then CRAG reduces to grading + abstention, which is score thresholding & abstention plus a query rewrite; implement that simpler thing.
- Low-latency paths where per-document grading is unaffordable — use reranker-score thresholds instead.
- When retrieval is simply weak: fix hybrid/reranking/chunking first. CRAG is a safety net, not a retrieval fix.
Production at scale
Use the reranker as the grader wherever possible. You already compute a calibrated relevance score per passage (reranking). Mapping score bands to Correct/Ambiguous/Incorrect gets most of CRAG's benefit at zero extra latency. Reserve LLM grading for the ambiguous band only.
Parallelise grading across documents with a bounded concurrency, and cap it (grade the top 6, not the top 50).
Web fallback governance. Explicit per-tenant setting; queries sanitised (never send raw internal text or identifiers); results clearly labelled as external in the answer and in citations (citations); domain allow-lists for regulated tenants; and treat web content as untrusted data with the full injection defences (prompt-injection defence, delimiters: instructions vs data).
Strip refinement pays for itself in token cost when passages are long (parents, parent–child). Measure tokens saved vs grading cost; on long-context pipelines it's usually net positive.
Branch distribution as an SLI. Track correct / ambiguous / incorrect rates per tenant and per query type, weekly. A shift toward "incorrect" means corpus drift, an ingestion failure, or a new question type — often before users complain.
Bound corrections to one round. A second rewrite-and-retry rarely helps and doubles the worst-case latency. If one correction doesn't produce usable evidence, abstain.
Provenance in the answer. When evidence came from the fallback path, say so: "I couldn't find this in internal documents; based on public sources…". This is a correctness and a trust requirement.
Evaluate the grader itself against human labels on a sample (LLM-as-judge) — grader precision/recall is the component that determines whether CRAG helps or hurts.
Prod checklist
- Reranker score as the primary grader; LLM grading only for the ambiguous band
- Grading parallelised and capped to top-N documents
- Strip refinement with a fallback to full passage when no strips are kept
- Exactly one correction round; then answer or abstain
- Web/secondary fallback gated per tenant; queries sanitised; results labelled external and treated as untrusted
- Branch (correct/ambiguous/incorrect) logged per query; weekly distribution dashboard
- Answer states when evidence came from the fallback path
- Grader accuracy measured against human labels on a sample
Metrics to watch
| Metric | Why |
|---|---|
| Branch distribution over time and per tenant | Retrieval/corpus health |
| Grader precision/recall vs human labels | Whether the safety net works |
| Tokens saved by strip refinement vs grading cost | Net economics |
| Faithfulness on ambiguous/incorrect branches | Fallback answer quality |
Related
- Score thresholding — the simpler ancestor
- Reranking — the cheap grader
- Self-RAG — model-driven rather than pipeline-driven correction
- Faithfulness check — the post-generation counterpart
- Injection — web fallback risk