Retrieval & ranking → Score thresholding & abstention
What it is
Vector search always returns k results. Ask a question your corpus can't answer and you still get five passages — the five least irrelevant ones — and a model dutifully constructing an answer from them. This is the mechanism behind "confidently wrong on a fresh question" (common failure modes).
Thresholding adds a decision before generation:
rerank top score ≥ τ_high → answer normally
τ_low ≤ top score < τ_high → answer with hedging / escalate (routing & escalation) / ask to clarify
top score < τ_low → abstain: "I don't have information about X"
+ suggest where to look / who to ask
The signal should come from the reranker (reranking), whose scores are calibrated relevance judgements, not from raw cosine similarity (which has no absolute meaning and drifts by corpus and query length) and not from RRF ranks (which discard magnitude entirely, hybrid search).
Abstention is a designed path with its own prompt, its own UX, and its own metrics — not an accident of the model deciding to hedge.
How to implement
- Choose the signal: reranker top score, or a small ensemble (top score + mean of top-3 + number of candidates above a mid-threshold). Log all of them from day one.
- Build a labelled set for calibration: golden questions the corpus can answer, plus an equal number of plausible questions it cannot (out-of-scope, out-of-date, about a different tenant's domain). The second half is the part everyone skips and the reason thresholds are usually wrong.
- Sweep τ and plot correct-abstention rate against false-refusal rate. Pick the operating point your product wants — a compliance assistant tolerates false refusals, a sales-enablement bot doesn't.
- Write the abstention response: state what was searched, that nothing sufficient was found, and offer a next step (a person, a system, a rephrase). Never a bare "I don't know".
- Verify after generation too: even above the threshold, a faithfulness check (faithfulness check) can catch answers unsupported by the retrieved text; the two controls are complementary.
- Escalate in the middle band rather than guessing: a stronger model, HyDE retry (hyDE), relaxed filters (self-query), or a clarifying question.
- Recalibrate on every reranker change, embedding change, or corpus expansion.
Small example
THRESHOLDS = {"tau_low": 0.25, "tau_high": 0.55} # calibrated per corpus + reranker version
def retrieval_confidence(reranked: list[dict]) -> dict:
if not reranked:
return {"top": 0.0, "mean3": 0.0, "n_strong": 0}
scores = [c["rerank_score"] for c in reranked]
return {"top": scores[0], "mean3": float(np.mean(scores[:3])),
"n_strong": sum(s >= THRESHOLDS["tau_high"] for s in scores)}
ABSTAIN_TEMPLATE = ("I couldn't find information about {topic} in the {corpus_name}. "
"I searched {n} documents covering {scope}. "
"You may want to check {suggestion} or ask {owner}.")
def answer_with_abstention(query, caller, corpus_meta):
candidates = hybrid_retrieve(query, caller)
reranked = rerank_hosted(query, candidates, top_n=8)
conf = retrieval_confidence(reranked)
log_confidence(query, conf)
if conf["top"] < THRESHOLDS["tau_low"]:
return {"answer": ABSTAIN_TEMPLATE.format(topic=summarise_topic(query), corpus_name=corpus_meta["name"],
n=corpus_meta["doc_count"], scope=corpus_meta["scope"],
suggestion=corpus_meta["fallback_system"], owner=corpus_meta["owner"]),
"abstained": True, "confidence": conf}
if conf["top"] < THRESHOLDS["tau_high"]:
retry = retrieve_with_hyde(query, caller) # one bounded recovery attempt
reranked2 = rerank_hosted(query, retry, top_n=8)
if retrieval_confidence(reranked2)["top"] > conf["top"]:
reranked, conf = reranked2, retrieval_confidence(reranked2)
if conf["top"] < THRESHOLDS["tau_low"]:
return {"answer": ABSTAIN_TEMPLATE.format(...), "abstained": True, "confidence": conf}
answer = generate(query, reranked, hedge=(conf["top"] < THRESHOLDS["tau_high"]))
return {"answer": answer, "abstained": False, "confidence": conf}
# Calibration
def sweep_thresholds(answerable, unanswerable, taus=np.arange(0.1, 0.8, 0.05)):
rows = []
for tau in taus:
fr = np.mean([retrieval_confidence(rerank_hosted(q, hybrid_retrieve(q, SYS_CALLER)))["top"] < tau
for q in answerable]) # false refusal
ca = np.mean([retrieval_confidence(rerank_hosted(q, hybrid_retrieve(q, SYS_CALLER)))["top"] < tau
for q in unanswerable]) # correct abstention
rows.append({"tau": round(float(tau), 2), "false_refusal": fr, "correct_abstention": ca})
return rows
Pros
- Directly prevents the most damaging failure: a confident, well-cited-looking answer built from irrelevant passages.
- Builds trust — users learn the system says when it doesn't know, which makes them trust the answers it does give.
- Cheap: a comparison on a score you already computed.
- Gives an escalation trigger for the middle band (routing & escalation, hyDE).
- Reduces cost — abstained queries skip generation.
Cons
- False refusals frustrate users and are highly visible ("it's right there in the document!").
- Thresholds are fragile: they must be recalibrated on every reranker, embedder, or corpus change, and they drift as the corpus grows.
- Score distributions vary by query type — long analytical questions score differently from short lookups; one global τ is a compromise.
- Abstaining when retrieval was merely weak (rather than the answer being absent) hides a retrieval bug behind a polite message.
When to use
- Every production RAG system. Especially high-stakes domains where a wrong answer costs more than no answer: legal, medical, financial, compliance, security.
- Any system whose corpus has clear boundaries users will cross ("what's the weather", "what's our stock price").
When NOT to use (aggressive thresholds)
- Exploratory/research assistants where users prefer partial signal over silence — use hedging and low-confidence labelling instead of refusal.
- Before you have a reranker: raw cosine thresholds are unreliable and will produce arbitrary refusals.
- As a substitute for fixing retrieval: if abstention rate is high, look at recall before tuning τ.
Production at scale
Measure the metric nobody measures. From the source file: correct-abstention rate and false-refusal rate are "the most-skipped RAG metric" (metrics cheat sheet). Put both on the dashboard, from launch. You need the unanswerable half of the eval set to compute them, so build it.
Per-query-type thresholds once you have routing or query classification (routing): factoid lookups, analytical questions, and list questions have different score distributions. Start global, split when the data justifies it.
Abstention quality is a UX problem. "I don't have that information" is a dead end. Say what was searched, what the corpus covers, and where to go next. Include a "search anyway" affordance for the user who knows better than your threshold — and log when they use it, because that's labelled data for recalibration.
Distinguish absent from inaccessible. If the ACL filter (tenant / ACL capture at ingest) removed the answer, the honest message is different — and you generally must not reveal that a document exists. Decide the policy deliberately: usually "I couldn't find information you have access to" without confirming existence.
Recalibration as a release gate. Any change to the reranker, embedder, chunking, or corpus scope triggers a threshold recalibration run in CI, with the operating point reviewed (the testing pyramid, versioning & rollout).
Drift monitoring. Track the distribution of top scores in production weekly. A downward shift means corpus/query drift; the threshold that was right in March silently over-refuses in September.
Pair with faithfulness checking. Thresholding catches "nothing relevant was retrieved"; faithfulness catches "relevant text was retrieved but the answer isn't supported by it" (faithfulness check). Both are needed; neither substitutes for the other.
Cost note. Abstaining early saves the generation call. In systems with a high out-of-scope rate (public-facing bots), this is a real line item.
Prod checklist
- Reranker score (not cosine, not RRF rank) as the confidence signal; all signals logged
- Calibration set with answerable and unanswerable questions
- τ_low / τ_high swept and chosen against the false-refusal / correct-abstention trade-off
- Middle band escalates (stronger model, HyDE retry, clarify) before refusing
- Abstention message states scope and next step; "search anyway" affordance logged
- ACL-filtered-out case handled with a distinct, non-disclosing message
- Recalibration gate on reranker/embedder/corpus changes
- Weekly top-score distribution drift monitoring
- Correct-abstention and false-refusal rates on the main dashboard
Metrics to watch
| Metric | Why |
|---|---|
| Correct-abstention rate | The point of the feature |
| False-refusal rate | The cost of the feature |
| Top-score distribution over time | Drift |
| "Search anyway" click rate | Threshold too strict |
| Abstention rate by query type / tenant | Coverage gaps |
Related
- Reranking — supplies the calibrated score
- Refusal / no-answer path — the generation-side design
- Faithfulness check — the complementary control
- Escalation — the middle band
- Golden dataset — must include unanswerable questions