Advanced RAG → Self-RAG (reflection tokens: retrieve? supported? useful?)

Self-RAG Explained: Reflection Tokens and Why to Validate Self-Grading

What it is

Self-RAG (Asai et al., 2023) trains a model to produce reflection tokens alongside its output:

TokenQuestion it answersValues
RetrieveDo I need to look something up for this segment?yes / no / continue
IsRelIs this retrieved passage relevant?relevant / irrelevant
IsSupIs this sentence I just wrote supported by the passage?fully / partially / no support
IsUseIs this response useful for the question?1–5

Generation becomes segment-by-segment: decide whether to retrieve, retrieve if needed, generate a segment, self-assess support, and (in the original method) beam-search over candidate continuations ranked by the reflection scores. The model can answer from parametric knowledge when retrieval isn't needed, and it self-filters unsupported claims as it writes.

In practice, most teams implement a prompted approximation rather than the trained model: a structured output where the model emits needs_retrieval, per-claim support labels, and citations, validated in code. You lose the trained calibration; you keep the architecture and can use any model.

How to implement (the practical version)

  1. Adaptive retrieval decision. A cheap first call (or the router, routing) outputs {"needs_retrieval": bool, "reason": ...}. Skip retrieval for chit-chat, formatting requests, and general knowledge — this is where much of Self-RAG's real-world value lies.
  2. Relevance self-check on retrieved passages — in practice, use the reranker score (reranking) rather than asking the model; it's cheaper and better calibrated.
  3. Structured generation with per-claim support: the model outputs an array of {claim, citations: [chunk_ids], support: "full"|"partial"|"none"} plus a final prose answer assembled from the claims.
  4. Validate in code: every cited ID must exist in the retrieved set; claims labelled none are dropped or flagged; if too many claims are unsupported, abstain (score thresholding & abstention).
  5. Optional usefulness pass: a separate cheap call scoring the assembled answer against the question; low scores trigger a retry or escalation (routing & escalation).
  6. Never trust the labels alone — sample-verify with an independent faithfulness check (faithfulness check); self-assessment is correlated with, not equal to, correctness.
  7. Log the reflection outputs — they're a rich per-claim quality signal for triage (feedback capture & triage).

Small example

SELF_RAG_SYS = """Answer the question using the passages. Work claim by claim.
For each factual claim you make, cite the passage ids that support it and label the support level.
Output JSON:
{
  "needs_more_retrieval": bool,
  "missing": string|null,
  "claims": [{"text": string, "citations": [string], "support": "full"|"partial"|"none"}],
  "answer": string,
  "usefulness": 1-5
}
Rules:
- support="full" only when a cited passage states the claim explicitly.
- Do not make claims you cannot cite; if the passages lack the information, say so in "missing".
- "answer" must contain only claims labelled full or partial."""

def self_rag_answer(question, caller, max_retrieval_rounds=2):
    evidence, rounds = [], 0
    needs = adaptive_retrieval_decision(question)                     # cheap call or router
    while rounds < max_retrieval_rounds:
        if needs:
            new = rerank_hosted(question, hybrid_retrieve(question, caller), top_n=6)
            evidence = mmr_select(evidence + new, embed_queries([question])[0], k=8)
        out = json.loads(llm("self_rag", SELF_RAG_SYS, [{"role": "user",
              "content": f"<question>{question}</question>\n<passages>{render(evidence)}</passages>"}],
              max_tokens=1200).text)

        valid_ids = {c["chunk_id"] for c in evidence}
        for cl in out["claims"]:                                       # code validation, not self-report alone
            bad = [c for c in cl["citations"] if c not in valid_ids]
            if bad: cl["support"] = "none"; cl["invalid_citations"] = bad
        unsupported = [c for c in out["claims"] if c["support"] == "none"]
        log_self_rag(question, n_claims=len(out["claims"]), n_unsupported=len(unsupported), usefulness=out["usefulness"])

        if out.get("needs_more_retrieval") and out.get("missing") and rounds + 1 < max_retrieval_rounds:
            question_aug, needs, rounds = out["missing"], True, rounds + 1
            continue
        break

    if not out["claims"] or len(unsupported) > len(out["claims"]) / 2:
        return abstain(question, evidence)
    return {"answer": out["answer"], "claims": [c for c in out["claims"] if c["support"] != "none"],
            "usefulness": out["usefulness"]}

Pros

  • Adaptive retrieval — skipping retrieval when it isn't needed saves latency and cost and avoids injecting irrelevant context into simple turns.
  • Per-claim support labels make hallucinations visible at the sentence level rather than the answer level.
  • Self-filtering removes unsupported claims before the user sees them.
  • Citations become structural, which makes validation in code straightforward (citations).
  • The reflection outputs are excellent triage data.

Cons

  • Self-assessment is not reliable. A model that hallucinated a claim will often also label it supported. Treat labels as a signal, verify independently.
  • The trained version requires a fine-tuned model with the reflection vocabulary; the prompted version is an approximation with weaker calibration.
  • Token cost: structured per-claim output is substantially more tokens than prose.
  • Latency: multiple passes if you add a usefulness check or a second retrieval round.
  • Segment-level beam search (the paper's full method) is impractical behind most inference APIs.

When to use

  • Mixed-traffic assistants where many turns don't need retrieval at all (adaptive retrieval is the biggest practical win).
  • High-stakes answers where per-claim citation and support labelling is a product requirement.
  • As a structured alternative to a separate post-hoc faithfulness pass, when you want the filtering inside generation.

When NOT to use

  • Latency-critical single-purpose Q&A — a simpler pipeline plus a post-hoc faithfulness check (faithfulness check) is cheaper and about as effective.
  • As your only hallucination control. Independent verification is required; self-labels are correlated, not conclusive.
  • When output token cost dominates your bill — per-claim JSON is expensive.

Production at scale

Take the adaptive-retrieval half, be sceptical of the self-grading half. In practice the reliable value of Self-RAG in production systems is the "should I retrieve at all" decision — often 20–40% of conversational turns — and the structured per-claim citation format. The self-support labels are useful as a cheap filter whose output you then verify on a sample.

Validate citations in code, always. Every cited ID must exist in the retrieved set; invalid citations force support: none. This single check catches most fabricated citations and costs nothing (citations).

Calibrate the labels. On a sample, compare model-assigned support against human or NLI judgement (faithfulness check). Publish the confusion matrix. If full means "actually supported" only 80% of the time, your product copy shouldn't say "verified".

Cost control: per-claim JSON roughly doubles output tokens versus prose. Use it on the high-stakes path only, or emit claims for the citation-bearing sentences and prose for the rest.

Combine, don't duplicate. If you already run CRAG-style grading (corrective RAG (CRAG): grade, then rewrite or fall back) before generation and a faithfulness check (faithfulness check) after, adding full Self-RAG in the middle is three overlapping controls. Pick the two that measure best on your eval set.

Abstention rule from claim labels: if more than half the claims are unsupported, abstain rather than presenting a filtered stub. Log it as a distinct abstention reason.

Reflection outputs into the flywheel (data flywheel): claims labelled partial or none are precisely the cases worth reviewing and turning into golden examples or hard negatives.

Prod checklist

  • Adaptive retrieval decision implemented (the highest-value half)
  • Structured per-claim output with citations and support labels
  • Citation IDs validated against the retrieved set in code; invalid → none
  • Abstention rule when unsupported claims dominate
  • Independent faithfulness verification on a sample; label calibration published
  • Bounded extra retrieval rounds (≤2)
  • Per-claim output reserved for high-stakes paths (token cost)
  • Reflection outputs routed into triage and golden-set growth

Metrics to watch

MetricWhy
Adaptive-retrieval skip rateCost/latency saved
Claim support-label distributionGeneration health
Label calibration (self-label vs verified)How much to trust it
Invalid-citation rateFabrication
Abstention rate from unsupported-claim ruleSafety net firing