Retrieval & ranking → Recursive / agentic retrieval (bounded follow-up searches)

Recursive Agentic Retrieval in RAG Explained: Bound the Loop

What it is

Single-shot retrieval commits to one query before seeing any evidence. But real research is iterative: you search, discover the actual term of art, discover a cross-reference ("as defined in Schedule 3"), notice the answer depends on a jurisdiction you now know to ask about — and search again.

Recursive retrieval gives the model that loop:

query → retrieve → model inspects results
                     ├── "sufficient" → generate answer
                     └── "insufficient: need X" → new query → retrieve → ...
                                                    (max N hops, budget-enforced)

It is the retrieval-specific, bounded form of agentic RAG (agentic RAG). The difference from decomposition (decomposition) is that sub-queries are formed after seeing evidence, not planned upfront — which handles cross-references, unknown terminology, and "it depends" answers that static decomposition can't anticipate.

How to implement

  1. Wrap retrieval as a tool the model can call with a query and optional filters (structured outputs & tool use, tool design): search(query, filters) → passages with ids.
  2. Loop with hard bounds: max_hops (2–4), max_total_passages, wall-clock timeout, token budget (cost & step budgets). The runtime enforces these, not the prompt.
  3. After each hop, ask for a structured decision: {"sufficient": bool, "missing": "...", "next_query": "...", "reason": "..."}. A structured stop decision is far more reliable than free-form "should I search again?".
  4. Accumulate evidence across hops with dedupe (MMR / diversity) and a per-hop budget (per-source truncation budgets), so hop 3 doesn't evict hop 1's decisive passage.
  5. Detect loops: if a proposed next query is near-identical to a previous one (cosine or normalised string match), stop — repeated identical searches is the classic runaway pattern.
  6. Stop on diminishing returns: if a hop adds no new chunk above the relevance threshold, stop.
  7. Always terminate with a decision: answer from what you have, or abstain (score thresholding & abstention). Never exit the loop silently.

Small example

DECIDE_SYS = """You are deciding whether the retrieved passages are sufficient to answer the question.
Output JSON: {"sufficient": bool, "missing": string, "next_query": string|null, "reason": string}
Set sufficient=true if the passages contain enough to answer completely.
If not, name exactly what is missing and give ONE focused follow-up search query.
Do not propose a query similar to one already tried."""

def recursive_retrieve(question, caller, max_hops=3, max_passages=20, deadline_s=8.0):
    t0, tried, evidence = time.time(), [], []
    query = question
    for hop in range(max_hops):
        if time.time() - t0 > deadline_s:
            log_hop_stop("deadline", hop); break
        tried.append(query)
        new = rerank_hosted(query, hybrid_retrieve(query, caller), top_n=6)
        new = [c for c in new if c["chunk_id"] not in {e["chunk_id"] for e in evidence}]
        if hop > 0 and not any(c["rerank_score"] >= THRESHOLDS["tau_high"] for c in new):
            log_hop_stop("no_new_relevant", hop); break                     # diminishing returns
        evidence = mmr_select(evidence + new, embed_queries([question])[0], k=max_passages)

        decision = json.loads(llm("hop_decide", DECIDE_SYS, [{"role": "user", "content":
            f"<question>{question}</question>\n<passages>{render(evidence)}</passages>\n<tried>{tried}</tried>"}],
            max_tokens=200).text)
        log_hop(hop, query, decision, n_evidence=len(evidence))
        if decision["sufficient"] or not decision.get("next_query"):
            break
        nq = decision["next_query"]
        if any(similar(nq, t) > 0.9 for t in tried):                        # loop guard
            log_hop_stop("loop_detected", hop); break
        query = nq

    conf = retrieval_confidence(evidence)
    if conf["top"] < THRESHOLDS["tau_low"]:
        return abstain(question, evidence)                                   # always a definite outcome
    return generate(question, evidence, hops=hop + 1)

Pros

  • Handles cross-references and unknown vocabulary — the model learns the corpus's term of art from hop 1 and uses it in hop 2.
  • Answers "it depends" questions by discovering the dependency and then resolving it.
  • Higher recall on genuinely hard questions than any single-shot strategy.
  • Explainable: the hop trail is a readable research path you can show the user.
  • Bounded version keeps the cost predictable, unlike open-ended agentic RAG.

Cons

  • Latency multiplies: each hop is a retrieval + rerank + decision call, 0.5–2 s each. Three hops is a slow answer.
  • Cost multiplies similarly.
  • Loops and drift: without guards, the model re-searches variants of the same thing, or wanders off-topic across hops.
  • Context accumulation: evidence from three hops competes for one window (per-source truncation budgets); without dedupe and budgets, later hops evict earlier decisive passages.
  • Error compounding: a wrong conclusion at hop 1 shapes hop 2's query (failure modes: poisoning, distraction, confusion, clash poisoning).
  • The stop decision is itself an LLM judgement and can be wrong in both directions.

When to use

  • Research-style questions over large, cross-referencing corpora: legal (defined terms, schedules), standards and regulations, technical documentation, incident investigation.
  • When single-shot + decomposition leaves a measurable gap on the golden set for a specific, identifiable query class.
  • Async or "deep research" modes where users accept 10–30 s for a thorough answer.

When NOT to use

  • Interactive chat with a sub-2-second latency budget — one hop maximum, or none.
  • Simple factoid corpora where hop 1 nearly always suffices (measure: what fraction of golden questions are answered at hop 1? If >90%, recursion is cost without benefit).
  • Before hybrid + reranking + decomposition are in place — those are cheaper and fix most of the same failures.
  • Unbounded. Ever. An unbounded retrieval loop is the "costs 10× the estimate" failure mode (common failure modes).

Production at scale

Two modes, explicitly. A fast interactive mode (0–1 hops) and an opt-in deep mode (up to 3–4 hops, async, with progress shown). Don't make every user pay research latency for a factoid question. Route by query complexity (routing) or by an explicit user action.

Budgets enforced by the runtime. Max hops, max passages, max tokens, max wall-clock, max dollars — checked in code before each hop, not requested in the prompt (cost & step budgets). Log which bound stopped the loop; the distribution tells you whether bounds are binding too often.

Loop and drift guards. Near-duplicate query detection (embedding cosine > 0.9 against all previous hops) and a topical-drift check (cosine between hop-N query and the original question below a floor → stop). Both are cheap and both prevent the expensive failure.

Evidence management across hops. Deduplicate by chunk ID, apply MMR (MMR / diversity), and keep a fixed evidence budget with the highest-reranked passages surviving. Never simply append.

Show the trail. For deep-research modes, streaming the hop queries as they happen ("searching for: definition of Schedule 3 terms…") turns a 20-second wait into an engaging one and gives users a way to correct course.

Attribute hops in evals. For each golden question, record the hop at which the decisive passage was found. This tells you your true max_hops (usually 2) and identifies query classes where recursion earns its cost (retrieval evals).

Cache across hops and requests: identical sub-queries recur; cache retrieval results by (query_hash, acl_signature) — with the ACL signature, or you've built a cross-tenant leak.

Failure containment. If a hop's retrieval or rerank fails, continue with existing evidence rather than aborting; degrade, don't crash (failure handling).

Prod checklist

  • Retrieval exposed as a bounded tool; hops capped (2–4) and enforced in the runtime
  • Structured stop decision (sufficient / missing / next_query), not free-form
  • Loop guard (near-duplicate queries) and drift guard (distance from original question)
  • Diminishing-returns stop when a hop adds no relevant new chunks
  • Evidence deduped, MMR-diversified, and budget-capped across hops
  • Fast mode vs deep mode; deep mode is opt-in/async with streamed hop trail
  • Always terminates in answer or abstention; stop reason logged
  • Retrieval cache keyed by query and ACL signature
  • Hop-of-decisive-passage recorded in evals

Metrics to watch

MetricWhy
Hop distribution and stop-reason distributionAre bounds binding? Are guards firing?
Answer quality by hop countWhether extra hops earn their cost
p95 latency and $/query in deep modeUnit economics
Loop-guard trigger rateModel drift/looping
Hop at which the decisive passage appeared (eval)Right max_hops