Query understanding → Multi-query + Reciprocal Rank Fusion
What it is
A single query embedding is one point in space; documents that answer the question but are phrased differently may sit just outside its neighbourhood. Multi-query asks an LLM for several rephrasings from different angles — formal/informal, different key terms, a hypothetical document title, a keyword-only version for BM25 — runs retrieval for each, and fuses.
Reciprocal Rank Fusion (RRF) merges ranked lists without needing comparable scores:
RRF(d) = Σ over lists L 1 / (k + rank_L(d)) (k ≈ 60)
A document that appears near the top of several lists scores highest; one that appears once at rank 1 still scores well; scores from dense (cosine) and sparse (BM25) don't need normalising. RRF is also the standard fusion for hybrid search (hybrid search) — multi-query simply adds more lists.
How to implement
- Generate variants (fast tier, temperature ~0.7 for diversity, or temperature 0 with explicit "give 4 different angles"): a rephrase, a keyword version, a version using likely document vocabulary, a "title of the document that answers this". Include the original.
- Run retrieval per variant in parallel — dense and/or BM25 — with the same ACL filters (filtered search); k per list ~20–30.
- Fuse with RRF to a single candidate list; keep per-document provenance (which variants hit it).
- Rerank the fused top-50 with a cross-encoder using the original query (reranking) — fusion is for recall, reranking is for precision.
- Dedupe by chunk ID (and by parent, parent–child).
- Budget: cap variants (4–5), cap total candidates, time-box the parallel retrieval.
- Evaluate: recall@k of fused vs single-query on the golden set; measure latency and cost added.
Small example
MQ_SYS = """Generate 4 alternative search queries for the user's question, each from a different angle:
1) a natural rephrasing, 2) a keyword-only version, 3) a version using formal/policy vocabulary, 4) a likely document section title.
Output JSON: {"queries": [string, string, string, string]}"""
def multi_query(q: str) -> list[str]:
out = llm("multi_query", MQ_SYS, [{"role": "user", "content": q}], max_tokens=200, temperature=0.4)
try:
variants = json.loads(out.text)["queries"][:4]
except Exception:
variants = []
return [q] + [v for v in variants if v and v.lower() != q.lower()]
def rrf(lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
scores = {}
for lst in lists:
for rank, doc_id in enumerate(lst, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores.items(), key=lambda x: -x[1])
async def retrieve_multi(q: str, caller, k_each: int = 25, k_out: int = 50):
variants = multi_query(q)
flt = build_acl_filter(caller)
dense = await asyncio.gather(*[vec.search_async(embed_queries([v])[0], k_each, flt) for v in variants])
sparse = await asyncio.gather(*[bm25.search_async(v, k_each, flt) for v in variants])
lists = [[h.id for h in l] for l in (*dense, *sparse)]
fused = rrf(lists)[:k_out]
log_multi_query(q, variants, {d: sum(d in l for l in lists) for d, _ in fused[:10]})
return [chunk_store.get(d) | {"rrf": s} for d, s in fused]
# then: reranker.rerank(original_query, candidates)[:8]
Pros
- Recall boost with minimal engineering; especially strong on short or ambiguous queries and vocabulary mismatch.
- RRF is robust: no score normalisation, works across dense/sparse/variants.
- Parallel retrieval keeps latency near single-query if the store handles concurrency.
- Provenance ("hit by 3 of 5 variants") is a useful confidence signal.
Cons
- Cost: one LLM call + n× retrieval per query; more candidates to rerank.
- Latency: the variant generation is serial before retrieval (~100–300 ms).
- Drift: variants can wander from the user's intent ("renewal term" → "contract length" → "duration of employment"); reranking with the original query mitigates.
- Diminishing returns beyond 3–5 variants; and no help when the corpus lacks the answer.
- RRF ignores score magnitude — a very confident dense hit and a weak one at the same rank count the same.
When to use
- Short, underspecified, or jargon-mismatched queries; consumer-style search; corpora with inconsistent terminology.
- When single-query recall@20 on the golden set is the bottleneck and reranking is already in place.
- As the fusion method for hybrid search regardless (hybrid search).
When NOT to use
- Long, precise queries (already well-specified) — little gain, pure cost; gate by query length/ambiguity.
- Latency budgets that can't absorb the generation step; consider HyDE (hyDE) or just hybrid.
- Without a reranker: fusion adds noise that only reranking cleans up.
Production at scale
Gate it. Run multi-query only when a cheap signal says it's worth it: short query, low single-query top score (score thresholding & abstention), or a rewrite flagged as ambiguous. Most well-formed queries skip it.
Cache variants by normalised query (short TTL); popular queries repeat.
Parallel retrieval capacity: n variants × 2 indexes = 8–10 concurrent searches per request; size the vector store/BM25 cluster for it, or reduce n under load (load-shedding rule, capacity & limits).
Consistent filters on every list (filtered search); fuse only ACL-filtered lists.
Weighted RRF: give the original query's lists a higher weight (e.g. 1.5×) and BM25 a tenant-tuned weight; evaluate on the golden set. Keep k=60 unless evals say otherwise.
Rerank with the original: the cross-encoder must see the user's actual question, not a variant — variants are for recall only.
Observability: per-request variant count, per-list latency, fused candidate count, and "variant contribution" (which variant produced the eventual top-cited chunk). If the original query alone almost always wins, turn multi-query off for that traffic.
Evals: recall@20 fused vs single; answer quality end-to-end; cost/latency deltas; per-query-type breakdown.
Prod checklist
- Variant generation on fast tier with a fixed count (≤ 5) and cached prompt
- Gating by query length/ambiguity/low top score
- Parallel retrieval with identical ACL filters per list; time-boxed
- RRF fusion (k=60, optional weights) → dedupe → rerank with original query
- Variant cache by normalised query
- Load-shedding rule to reduce n under pressure
- Golden-set recall and cost/latency deltas tracked; variant contribution logged
Metrics to watch
| Metric | Why |
|---|---|
| recall@20 fused vs single-query | Value |
| Added latency p95; retrieval QPS multiplier | Cost |
| Variant contribution to top-cited chunk | Whether variants matter |
| Gating rate | Discipline |
Related
- Query rewriting — precedes this
- HyDE — a different kind of variant
- Hybrid search — RRF is the fusion there too
- Reranking — required after fusion
- Score thresholding — gating signal