Generation & grounding → Refusal / no-answer path
Video takeaway
"Most RAG systems are never measured on whether they correctly decline — and that's the metric that decides whether anyone trusts them. Design the no-answer path like a feature: name the reason, say what you searched and what that corpus covers, answer whatever part you can, and give a real next step — the team to ask, the system to try. Make permission refusals reveal nothing about whether the document exists. Give users a 'search anyway' button, because every click on it is free evidence your threshold is too strict. And track two numbers forever: how often you correctly say you don't know, and how often you refuse something you actually had."
What it is
Every RAG system needs an answer for questions it cannot answer:
| Situation | Right response |
|---|---|
| Corpus has nothing on the topic | "I couldn't find anything about X in [corpus]. Try [system] or ask [owner]." |
| Retrieved passages are weakly relevant | Answer the part you can, name what's missing |
| Caller lacks permission to the answer | "I couldn't find information you have access to" — without confirming the document exists |
| Question needs a backend you don't have | "I can't query live billing data" — specific limitation, not a generic shrug |
| Faithfulness check failed | Deliver only the supported subset, or decline |
| Question is out of scope by design | State the scope and redirect |
| Document was processed but degraded (bounded escalation with visible failure) | Answer with a note that part of the source could not be read |
The failure this prevents: a system that always produces something teaches users that its answers are decorative. The failure it introduces: over-refusal, which teaches users the system is useless. Both are measurable; most teams measure neither.
Refusal decisions come from several places — retrieval score thresholds (score thresholding & abstention), faithfulness failure (faithfulness check), routing to a missing backend (routing), guardrails (input guardrails/08.3) — but they should converge on one refusal renderer so the experience is consistent and the metrics are aggregatable.
How to implement
- Enumerate refusal reasons as an enum:
no_relevant_results,low_confidence,unsupported_claims,no_permission,no_backend,out_of_scope,degraded_source,policy_blocked. - One renderer that takes
(reason, context)and produces a response with: what was searched, what wasn't found, what is available nearby, and a concrete next step. - Partial answers by default. If any part of the question is answerable, answer it and name the gap — a complete refusal to a partly answerable question is the most annoying failure mode.
- Never confirm existence for permission-based refusals; use a formulation that's identical whether the document doesn't exist or the caller can't see it.
- Offer next steps: the owning team, an adjacent system, a rephrase suggestion, or a "request access" action.
- Escalation before refusal: try HyDE (hyDE), relaxed non-security filters (self-query), or a stronger model (routing & escalation) once, then refuse.
- Instrument: every refusal logs its reason, the query, the confidence signals, and whether the user retried or rephrased.
Small example
from enum import Enum
class RefusalReason(str, Enum):
NO_RELEVANT = "no_relevant_results"; LOW_CONF = "low_confidence"
UNSUPPORTED = "unsupported_claims"; NO_PERMISSION = "no_permission"
NO_BACKEND = "no_backend"; OUT_OF_SCOPE = "out_of_scope"
DEGRADED = "degraded_source"; POLICY = "policy_blocked"
TEMPLATES = {
RefusalReason.NO_RELEVANT: ("I couldn't find anything about {topic} in {corpus}. "
"That collection covers {scope}. You could try {alt_system} or ask {owner}."),
RefusalReason.LOW_CONF: ("I found some related material on {topic}, but nothing that clearly answers your question. "
"The closest documents were {near_titles}. Want me to search those in full?"),
RefusalReason.UNSUPPORTED: ("I can confirm: {supported}. I couldn't verify the rest of your question against the sources, "
"so I'd rather not guess."),
# identical whether the document is absent or inaccessible — never reveals existence
RefusalReason.NO_PERMISSION: ("I couldn't find information about {topic} that you have access to. "
"If you believe you should have access, request it via {access_path}."),
RefusalReason.NO_BACKEND: ("I can search documents, but I can't query {backend} yet. For that, use {alt_system}."),
RefusalReason.OUT_OF_SCOPE:("I only cover {scope}. Your question is about {topic}, which is outside that."),
RefusalReason.DEGRADED: ("I answered from the readable parts of {doc}, but {n} pages of it could not be processed, "
"so this may be incomplete."),
}
def refuse(reason: RefusalReason, ctx: dict, partial: dict | None = None):
body = TEMPLATES[reason].format(**ctx)
resp = {"refused": True, "reason": reason.value, "message": body,
"next_steps": ctx.get("next_steps", []), "partial_answer": partial}
if partial and partial.get("claims"): # answer what you can, always
resp["message"] = f"{partial['answer']}\n\nWhat I couldn't determine: {body}"
resp["refused"] = "partial"
log_refusal(reason.value, ctx.get("query"), ctx.get("confidence"))
return resp
def answer_or_refuse(question, caller, corpus_meta):
route = route_question(question, caller)
if route["route"] not in AVAILABLE_BACKENDS:
return refuse(RefusalReason.NO_BACKEND, {"backend": route["route"], "alt_system": corpus_meta["bi_tool"], "query": question})
chunks = rerank_hosted(question, hybrid_retrieve(question, caller), top_n=8)
conf = retrieval_confidence(chunks)
if conf["top"] < THRESHOLDS["tau_low"]:
if not chunks and had_acl_filtered_results(question, caller): # existence never confirmed
return refuse(RefusalReason.NO_PERMISSION, {"topic": topic_of(question), "access_path": corpus_meta["access_path"], "query": question})
return refuse(RefusalReason.NO_RELEVANT, {"topic": topic_of(question), "corpus": corpus_meta["name"],
"scope": corpus_meta["scope"], "alt_system": corpus_meta["alt"],
"owner": corpus_meta["owner"], "query": question, "confidence": conf})
resp = generate_grounded(question, chunks)
gated = faithfulness_gate(resp, chunks)
if gated.get("abstained"):
return refuse(RefusalReason.UNSUPPORTED, {"supported": summarise(gated["supported"]), "query": question}, partial=gated)
if any(c.get("degraded_pages") for c in chunks):
gated["note"] = TEMPLATES[RefusalReason.DEGRADED].format(doc=chunks[0]["title"], n=chunks[0]["degraded_pages"])
return gated
Pros
- Trust: a system that declines credibly is believed when it answers. This is the single biggest driver of adoption in enterprise deployments.
- Prevents the most damaging failure — a confident, well-formatted, wrong answer.
- Actionable refusals ("ask the benefits team") are genuinely useful outcomes, not dead ends.
- Refusal telemetry maps corpus gaps — the highest-value input to knowing what to ingest next.
- Cheaper: refused queries skip generation.
Cons
- False refusals are highly visible and infuriating — "it's right there in the document".
- Users may not distinguish "not in the corpus" from "doesn't exist", and act on the wrong conclusion; wording matters.
- Partial answers are harder to write than either a full answer or a clean refusal.
- Permission-based refusals must be vague, which is inherently unsatisfying — and necessarily so.
- Requires the unanswerable half of the eval set, which most teams never build.
When to use
Every production RAG system. The refusal path is not optional; the only choice is whether it's designed or accidental.
When NOT to use (aggressively)
- Exploratory research assistants where users prefer weak signal over silence — surface low-confidence results with explicit hedging instead of refusing.
- Internal tools with expert users who can judge relevance themselves — lower the threshold and label confidence.
Production at scale
Measure the two rates forever (metrics cheat sheet): correct-abstention rate and false-refusal rate. Both require an eval set containing questions the corpus genuinely cannot answer — build it, keep it current as the corpus grows, and re-run it on every retrieval or model change.
One renderer, many callers. Thresholds, faithfulness gates, guardrails, and routing all funnel into the same refusal function so wording is consistent and reasons are comparable in the metrics.
"Search anyway" as an escape hatch. Let users override a low-confidence refusal. Every click is a labelled data point saying your threshold was too strict — one of the cheapest sources of calibration data you'll get.
Permission refusals must be information-free. Identical message whether the document is absent or inaccessible; identical latency where feasible. Otherwise the refusal itself becomes an existence oracle (authentication & authorization).
Refusal reasons drive different owners: no_relevant → content/ingestion team (fill the gap); no_backend → product roadmap; no_permission → access management; unsupported → retrieval/generation quality. Route the telemetry accordingly (feedback capture & triage).
Refusal rate by tenant and query class is a coverage map. A tenant with a 40% refusal rate has an ingestion problem; a query class with a high rate is an unserved use case.
Never fake capability. If the system can't do something, say what it can't do specifically. "I can't query live billing" is useful; "I don't have that information" is not.
Degraded-source notes close the loop with the ingestion retry ladder (bounded escalation with visible failure): when a cited document has unprocessed pages, the user is told, which is the visible-failure principle carried all the way to the answer.
Prod checklist
- Refusal reasons enumerated; single renderer used by all callers
- Partial answers preferred over total refusal whenever any part is answerable
- Permission refusals reveal no existence information; wording and latency uniform
- Concrete next steps (owner, alternative system, access request) in every refusal
- One escalation attempt (HyDE / relaxed filters / stronger model) before refusing
- "Search anyway" affordance; overrides logged as calibration data
- Unanswerable questions in the golden set; correct-abstention and false-refusal rates on the dashboard
- Refusal reason telemetry routed to the owning team
- Degraded-source notes surfaced from ingestion through to the answer
Metrics to watch
| Metric | Why |
|---|---|
| Correct-abstention rate | The most-skipped RAG metric |
| False-refusal rate | The cost of abstaining |
| Refusal rate by reason, tenant, query class | Coverage map and ownership routing |
| "Search anyway" override rate | Threshold calibration |
| Post-refusal user behaviour (rephrase / abandon / escalate) | Whether the refusal was useful |
Related
- Score thresholding — the main trigger
- Faithfulness — the unsupported-claims trigger
- Routing — the missing-backend trigger
- Retry ladder — degraded-source notes
- AuthZ — non-disclosing permission refusals
- Golden dataset — must contain unanswerable questions