Retrieval & ranking → Recursive / agentic retrieval (bounded follow-up searches)
What it is
Single-shot retrieval commits to one query before seeing any evidence. But real research is iterative: you search, discover the actual term of art, discover a cross-reference ("as defined in Schedule 3"), notice the answer depends on a jurisdiction you now know to ask about — and search again.
Recursive retrieval gives the model that loop:
query → retrieve → model inspects results
├── "sufficient" → generate answer
└── "insufficient: need X" → new query → retrieve → ...
(max N hops, budget-enforced)
It is the retrieval-specific, bounded form of agentic RAG (agentic RAG). The difference from decomposition (decomposition) is that sub-queries are formed after seeing evidence, not planned upfront — which handles cross-references, unknown terminology, and "it depends" answers that static decomposition can't anticipate.
How to implement
- Wrap retrieval as a tool the model can call with a query and optional filters (structured outputs & tool use, tool design):
search(query, filters) → passages with ids. - Loop with hard bounds:
max_hops(2–4),max_total_passages, wall-clock timeout, token budget (cost & step budgets). The runtime enforces these, not the prompt. - After each hop, ask for a structured decision:
{"sufficient": bool, "missing": "...", "next_query": "...", "reason": "..."}. A structured stop decision is far more reliable than free-form "should I search again?". - Accumulate evidence across hops with dedupe (MMR / diversity) and a per-hop budget (per-source truncation budgets), so hop 3 doesn't evict hop 1's decisive passage.
- Detect loops: if a proposed next query is near-identical to a previous one (cosine or normalised string match), stop — repeated identical searches is the classic runaway pattern.
- Stop on diminishing returns: if a hop adds no new chunk above the relevance threshold, stop.
- Always terminate with a decision: answer from what you have, or abstain (score thresholding & abstention). Never exit the loop silently.
Small example
DECIDE_SYS = """You are deciding whether the retrieved passages are sufficient to answer the question.
Output JSON: {"sufficient": bool, "missing": string, "next_query": string|null, "reason": string}
Set sufficient=true if the passages contain enough to answer completely.
If not, name exactly what is missing and give ONE focused follow-up search query.
Do not propose a query similar to one already tried."""
def recursive_retrieve(question, caller, max_hops=3, max_passages=20, deadline_s=8.0):
t0, tried, evidence = time.time(), [], []
query = question
for hop in range(max_hops):
if time.time() - t0 > deadline_s:
log_hop_stop("deadline", hop); break
tried.append(query)
new = rerank_hosted(query, hybrid_retrieve(query, caller), top_n=6)
new = [c for c in new if c["chunk_id"] not in {e["chunk_id"] for e in evidence}]
if hop > 0 and not any(c["rerank_score"] >= THRESHOLDS["tau_high"] for c in new):
log_hop_stop("no_new_relevant", hop); break # diminishing returns
evidence = mmr_select(evidence + new, embed_queries([question])[0], k=max_passages)
decision = json.loads(llm("hop_decide", DECIDE_SYS, [{"role": "user", "content":
f"<question>{question}</question>\n<passages>{render(evidence)}</passages>\n<tried>{tried}</tried>"}],
max_tokens=200).text)
log_hop(hop, query, decision, n_evidence=len(evidence))
if decision["sufficient"] or not decision.get("next_query"):
break
nq = decision["next_query"]
if any(similar(nq, t) > 0.9 for t in tried): # loop guard
log_hop_stop("loop_detected", hop); break
query = nq
conf = retrieval_confidence(evidence)
if conf["top"] < THRESHOLDS["tau_low"]:
return abstain(question, evidence) # always a definite outcome
return generate(question, evidence, hops=hop + 1)
Pros
- Handles cross-references and unknown vocabulary — the model learns the corpus's term of art from hop 1 and uses it in hop 2.
- Answers "it depends" questions by discovering the dependency and then resolving it.
- Higher recall on genuinely hard questions than any single-shot strategy.
- Explainable: the hop trail is a readable research path you can show the user.
- Bounded version keeps the cost predictable, unlike open-ended agentic RAG.
Cons
- Latency multiplies: each hop is a retrieval + rerank + decision call, 0.5–2 s each. Three hops is a slow answer.
- Cost multiplies similarly.
- Loops and drift: without guards, the model re-searches variants of the same thing, or wanders off-topic across hops.
- Context accumulation: evidence from three hops competes for one window (per-source truncation budgets); without dedupe and budgets, later hops evict earlier decisive passages.
- Error compounding: a wrong conclusion at hop 1 shapes hop 2's query (failure modes: poisoning, distraction, confusion, clash poisoning).
- The stop decision is itself an LLM judgement and can be wrong in both directions.
When to use
- Research-style questions over large, cross-referencing corpora: legal (defined terms, schedules), standards and regulations, technical documentation, incident investigation.
- When single-shot + decomposition leaves a measurable gap on the golden set for a specific, identifiable query class.
- Async or "deep research" modes where users accept 10–30 s for a thorough answer.
When NOT to use
- Interactive chat with a sub-2-second latency budget — one hop maximum, or none.
- Simple factoid corpora where hop 1 nearly always suffices (measure: what fraction of golden questions are answered at hop 1? If >90%, recursion is cost without benefit).
- Before hybrid + reranking + decomposition are in place — those are cheaper and fix most of the same failures.
- Unbounded. Ever. An unbounded retrieval loop is the "costs 10× the estimate" failure mode (common failure modes).
Production at scale
Two modes, explicitly. A fast interactive mode (0–1 hops) and an opt-in deep mode (up to 3–4 hops, async, with progress shown). Don't make every user pay research latency for a factoid question. Route by query complexity (routing) or by an explicit user action.
Budgets enforced by the runtime. Max hops, max passages, max tokens, max wall-clock, max dollars — checked in code before each hop, not requested in the prompt (cost & step budgets). Log which bound stopped the loop; the distribution tells you whether bounds are binding too often.
Loop and drift guards. Near-duplicate query detection (embedding cosine > 0.9 against all previous hops) and a topical-drift check (cosine between hop-N query and the original question below a floor → stop). Both are cheap and both prevent the expensive failure.
Evidence management across hops. Deduplicate by chunk ID, apply MMR (MMR / diversity), and keep a fixed evidence budget with the highest-reranked passages surviving. Never simply append.
Show the trail. For deep-research modes, streaming the hop queries as they happen ("searching for: definition of Schedule 3 terms…") turns a 20-second wait into an engaging one and gives users a way to correct course.
Attribute hops in evals. For each golden question, record the hop at which the decisive passage was found. This tells you your true max_hops (usually 2) and identifies query classes where recursion earns its cost (retrieval evals).
Cache across hops and requests: identical sub-queries recur; cache retrieval results by (query_hash, acl_signature) — with the ACL signature, or you've built a cross-tenant leak.
Failure containment. If a hop's retrieval or rerank fails, continue with existing evidence rather than aborting; degrade, don't crash (failure handling).
Prod checklist
- Retrieval exposed as a bounded tool; hops capped (2–4) and enforced in the runtime
- Structured stop decision (
sufficient/missing/next_query), not free-form - Loop guard (near-duplicate queries) and drift guard (distance from original question)
- Diminishing-returns stop when a hop adds no relevant new chunks
- Evidence deduped, MMR-diversified, and budget-capped across hops
- Fast mode vs deep mode; deep mode is opt-in/async with streamed hop trail
- Always terminates in answer or abstention; stop reason logged
- Retrieval cache keyed by query and ACL signature
- Hop-of-decisive-passage recorded in evals
Metrics to watch
| Metric | Why |
|---|---|
| Hop distribution and stop-reason distribution | Are bounds binding? Are guards firing? |
| Answer quality by hop count | Whether extra hops earn their cost |
| p95 latency and $/query in deep mode | Unit economics |
| Loop-guard trigger rate | Model drift/looping |
| Hop at which the decisive passage appeared (eval) | Right max_hops |
Related
- Decomposition — planned upfront vs discovered
- Agentic RAG — the unbounded, tool-driven generalisation
- Cost & step budgets — enforcement
- Abstention — the terminal decision
- Context poisoning — error compounding across hops