Query understanding → Multi-query + Reciprocal Rank Fusion

Multi-Query Retrieval + Reciprocal Rank Fusion (RRF) Explained

What it is

A single query embedding is one point in space; documents that answer the question but are phrased differently may sit just outside its neighbourhood. Multi-query asks an LLM for several rephrasings from different angles — formal/informal, different key terms, a hypothetical document title, a keyword-only version for BM25 — runs retrieval for each, and fuses.

Reciprocal Rank Fusion (RRF) merges ranked lists without needing comparable scores:

RRF(d) = Σ over lists L  1 / (k + rank_L(d))       (k ≈ 60)

A document that appears near the top of several lists scores highest; one that appears once at rank 1 still scores well; scores from dense (cosine) and sparse (BM25) don't need normalising. RRF is also the standard fusion for hybrid search (hybrid search) — multi-query simply adds more lists.

How to implement

  1. Generate variants (fast tier, temperature ~0.7 for diversity, or temperature 0 with explicit "give 4 different angles"): a rephrase, a keyword version, a version using likely document vocabulary, a "title of the document that answers this". Include the original.
  2. Run retrieval per variant in parallel — dense and/or BM25 — with the same ACL filters (filtered search); k per list ~20–30.
  3. Fuse with RRF to a single candidate list; keep per-document provenance (which variants hit it).
  4. Rerank the fused top-50 with a cross-encoder using the original query (reranking) — fusion is for recall, reranking is for precision.
  5. Dedupe by chunk ID (and by parent, parent–child).
  6. Budget: cap variants (4–5), cap total candidates, time-box the parallel retrieval.
  7. Evaluate: recall@k of fused vs single-query on the golden set; measure latency and cost added.

Small example

MQ_SYS = """Generate 4 alternative search queries for the user's question, each from a different angle:
1) a natural rephrasing, 2) a keyword-only version, 3) a version using formal/policy vocabulary, 4) a likely document section title.
Output JSON: {"queries": [string, string, string, string]}"""

def multi_query(q: str) -> list[str]:
    out = llm("multi_query", MQ_SYS, [{"role": "user", "content": q}], max_tokens=200, temperature=0.4)
    try:
        variants = json.loads(out.text)["queries"][:4]
    except Exception:
        variants = []
    return [q] + [v for v in variants if v and v.lower() != q.lower()]

def rrf(lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores = {}
    for lst in lists:
        for rank, doc_id in enumerate(lst, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

async def retrieve_multi(q: str, caller, k_each: int = 25, k_out: int = 50):
    variants = multi_query(q)
    flt = build_acl_filter(caller)
    dense = await asyncio.gather(*[vec.search_async(embed_queries([v])[0], k_each, flt) for v in variants])
    sparse = await asyncio.gather(*[bm25.search_async(v, k_each, flt) for v in variants])
    lists = [[h.id for h in l] for l in (*dense, *sparse)]
    fused = rrf(lists)[:k_out]
    log_multi_query(q, variants, {d: sum(d in l for l in lists) for d, _ in fused[:10]})
    return [chunk_store.get(d) | {"rrf": s} for d, s in fused]
# then: reranker.rerank(original_query, candidates)[:8]

Pros

  • Recall boost with minimal engineering; especially strong on short or ambiguous queries and vocabulary mismatch.
  • RRF is robust: no score normalisation, works across dense/sparse/variants.
  • Parallel retrieval keeps latency near single-query if the store handles concurrency.
  • Provenance ("hit by 3 of 5 variants") is a useful confidence signal.

Cons

  • Cost: one LLM call + n× retrieval per query; more candidates to rerank.
  • Latency: the variant generation is serial before retrieval (~100–300 ms).
  • Drift: variants can wander from the user's intent ("renewal term" → "contract length" → "duration of employment"); reranking with the original query mitigates.
  • Diminishing returns beyond 3–5 variants; and no help when the corpus lacks the answer.
  • RRF ignores score magnitude — a very confident dense hit and a weak one at the same rank count the same.

When to use

  • Short, underspecified, or jargon-mismatched queries; consumer-style search; corpora with inconsistent terminology.
  • When single-query recall@20 on the golden set is the bottleneck and reranking is already in place.
  • As the fusion method for hybrid search regardless (hybrid search).

When NOT to use

  • Long, precise queries (already well-specified) — little gain, pure cost; gate by query length/ambiguity.
  • Latency budgets that can't absorb the generation step; consider HyDE (hyDE) or just hybrid.
  • Without a reranker: fusion adds noise that only reranking cleans up.

Production at scale

Gate it. Run multi-query only when a cheap signal says it's worth it: short query, low single-query top score (score thresholding & abstention), or a rewrite flagged as ambiguous. Most well-formed queries skip it.

Cache variants by normalised query (short TTL); popular queries repeat.

Parallel retrieval capacity: n variants × 2 indexes = 8–10 concurrent searches per request; size the vector store/BM25 cluster for it, or reduce n under load (load-shedding rule, capacity & limits).

Consistent filters on every list (filtered search); fuse only ACL-filtered lists.

Weighted RRF: give the original query's lists a higher weight (e.g. 1.5×) and BM25 a tenant-tuned weight; evaluate on the golden set. Keep k=60 unless evals say otherwise.

Rerank with the original: the cross-encoder must see the user's actual question, not a variant — variants are for recall only.

Observability: per-request variant count, per-list latency, fused candidate count, and "variant contribution" (which variant produced the eventual top-cited chunk). If the original query alone almost always wins, turn multi-query off for that traffic.

Evals: recall@20 fused vs single; answer quality end-to-end; cost/latency deltas; per-query-type breakdown.

Prod checklist

  • Variant generation on fast tier with a fixed count (≤ 5) and cached prompt
  • Gating by query length/ambiguity/low top score
  • Parallel retrieval with identical ACL filters per list; time-boxed
  • RRF fusion (k=60, optional weights) → dedupe → rerank with original query
  • Variant cache by normalised query
  • Load-shedding rule to reduce n under pressure
  • Golden-set recall and cost/latency deltas tracked; variant contribution logged

Metrics to watch

MetricWhy
recall@20 fused vs single-queryValue
Added latency p95; retrieval QPS multiplierCost
Variant contribution to top-cited chunkWhether variants matter
Gating rateDiscipline