Retrieval & ranking → Hybrid search (BM25 + dense, fused)
What it is
Two retrieval paradigms with complementary failure modes:
| BM25 / sparse | Dense / vector | |
|---|---|---|
| Matches | Exact tokens, weighted by rarity (IDF) and length-normalised | Semantic proximity in embedding space |
| Wins on | IDs (SKU-4471), names, acronyms, error codes, rare technical terms, exact phrases, negation-ish keyword presence | Paraphrase ("time off" ↔ "annual leave"), concepts, cross-lingual, typo tolerance |
| Fails on | Vocabulary mismatch — user says "vacation", doc says "annual leave entitlement" | Rare literals — SKU-4471 embeds like every other alphanumeric string |
| Needs | Inverted index, tokenizer, stemming/analyzers | Embedding model, ANN index |
Hybrid runs both and merges. Two fusion methods:
- RRF (rank-based):
score(d) = Σ 1/(k + rank_L(d)), k≈60. No score normalisation needed — the standard, robust default (multi-query + Reciprocal Rank Fusion). - Weighted score fusion: normalise each list's scores (min-max or z-score per query) and combine
α·dense + (1−α)·sparse. Tunable, slightly better when calibrated, but score distributions shift per query and per corpus, so it needs care.
Also relevant: learned sparse (SPLADE, BGE-M3's sparse head, Elastic ELSER) — neural models producing sparse term-weight vectors that expand a query with related terms while keeping exact-match behaviour. Best of both, at extra index and inference cost.
How to implement
- Build both indexes over the same chunk text — the indexed text including breadcrumbs and contextual headers (contextual retrieval), so both benefit from enrichment.
- Configure the analyzer for BM25: language-appropriate stemming, but preserve identifiers — don't strip hyphens, digits, or case-sensitive codes. Add a
keywordsub-field for exact-match on IDs. - Apply identical filters to both (tenant, ACL,
is_current) — fusing a filtered dense list with an unfiltered sparse list is a leak (filtered search). - Retrieve k≈25–50 from each, fuse with RRF to a candidate pool of ~50.
- Rerank the pool with a cross-encoder (reranking) and cut to 5–8. Fusion is for recall; reranking is for precision.
- Dedupe by chunk ID, then collapse to parents if using parent–child (parent–child).
- Evaluate: recall@50 for dense-only, sparse-only, and hybrid on the golden set, split by query type (identifier queries vs conceptual queries) — that split is where the argument for hybrid becomes visible.
Small example
def rrf(lists: dict[str, list[str]], weights: dict[str, float] | None = None, k: int = 60):
weights = weights or {name: 1.0 for name in lists}
scores, seen_in = {}, {}
for name, ids in lists.items():
for rank, doc_id in enumerate(ids, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + weights[name] / (k + rank)
seen_in.setdefault(doc_id, []).append(name)
return sorted(((d, s, seen_in[d]) for d, s in scores.items()), key=lambda x: -x[1])
def weighted_fusion(dense: list[tuple[str, float]], sparse: list[tuple[str, float]], alpha: float = 0.6):
def norm(pairs):
if not pairs: return {}
vals = [s for _, s in pairs]; lo, hi = min(vals), max(vals)
rng = (hi - lo) or 1.0
return {d: (s - lo) / rng for d, s in pairs}
dn, sn = norm(dense), norm(sparse)
ids = set(dn) | set(sn)
return sorted(((d, alpha * dn.get(d, 0) + (1 - alpha) * sn.get(d, 0)) for d in ids), key=lambda x: -x[1])
async def hybrid_retrieve(query: str, caller, k_each: int = 30, pool: int = 50):
flt = build_acl_filter(caller) # SAME filter on both legs — non-negotiable
qv = embed_queries([query])[0]
dense_hits, sparse_hits = await asyncio.gather(
vec.search_async(qv, k_each, flt),
bm25.search_async(query, k_each, flt),
)
fused = rrf({"dense": [h.id for h in dense_hits], "sparse": [h.id for h in sparse_hits]},
weights={"dense": 1.0, "sparse": 1.0})[:pool]
log_hybrid(query, n_dense_only=sum(1 for _, _, src in fused if src == ["dense"]),
n_sparse_only=sum(1 for _, _, src in fused if src == ["sparse"]),
n_both=sum(1 for _, _, src in fused if len(src) == 2))
return [chunk_store.get(d) | {"fusion_score": s, "matched_by": src} for d, s, src in fused]
Elasticsearch/OpenSearch and Qdrant/Weaviate/Milvus/Vespa can do both legs plus fusion server-side in one query — prefer that when available (one round trip, consistent filtering).
Pros
- Covers both failure modes — the single most reliable retrieval improvement after reranking.
- Identifiers, codes, and names become findable, which is the difference between a usable enterprise system and a frustrating one.
- RRF needs no tuning and is robust across corpora and query types.
- Cheap: BM25 is fast and inexpensive; no extra LLM calls.
- Gives a useful signal: chunks matched by both legs are usually the strongest candidates.
Cons
- Two indexes to build, keep in sync, migrate, and monitor. Text changes (contextual retrieval, boilerplate stripping) require re-indexing both.
- Analyzer configuration is fiddly and language-specific; wrong stemming breaks exact-match on IDs.
- RRF discards score magnitude — a strongly-matching document and a weak one at the same rank contribute equally, which hurts abstention decisions (score thresholding & abstention).
- Weighted fusion needs per-corpus tuning and can be unstable across query types.
- Latency is max(dense, sparse) if parallel, sum if not.
When to use
Essentially always for enterprise/document corpora. Especially when the corpus contains identifiers, product codes, names, legal citations, error codes, or domain acronyms — which is nearly all of them.
When NOT to use
- Purely conceptual corpora with no literals and consistent vocabulary (rare).
- Extremely latency-constrained paths where one leg must be dropped — drop dense before sparse for identifier-heavy traffic, and vice versa; measure which.
- When your store can't apply identical filters to both legs — fix that first, or you have a security bug rather than a retrieval feature.
Production at scale
Filter parity is a security property, not an optimisation. Write one filter-building function and pass its output to both legs. Add a test that asserts the sparse and dense queries carry the same filter object. This is the most common way a hybrid implementation leaks.
Index synchronisation. Both indexes are derived from the chunk registry (change detection). Write to both in the same ingestion transaction/queue message; monitor per-index chunk counts and alert on divergence. A chunk in dense but not sparse is a silently degraded result for identifier queries.
Analyzer tuning for your domain. Test: does SKU-4471 survive tokenisation? Does § 3.2 match? Does COVID-19 match covid19? Build a small analyzer test suite with your real identifiers; run it in CI. Add a keyword field for exact codes and boost it.
Learned sparse (SPLADE/ELSER) replaces the BM25 leg with a model that expands terms — typically a few points better than BM25 and it handles some vocabulary mismatch, at the cost of inference at both index and query time, and larger indexes. Evaluate on your golden set before adopting; the gain over BM25+dense+rerank is often small.
Weights per query type. If routing (routing) or a cheap classifier tells you the query is identifier-like, upweight sparse; if conceptual, upweight dense. Measurable gains, but keep it simple — RRF unweighted is a strong baseline that rarely justifies replacing.
Match-source telemetry. Log how many final-cited chunks were found by dense only, sparse only, or both. If sparse-only contributions are near zero, your corpus may not need hybrid (rare); if they're 20%+, hybrid is carrying real weight and you can quantify what removing it would cost.
Cost. BM25 is cheap; the real cost of hybrid is the second index's memory/storage and the extra reranking candidates. Budget the pool size deliberately.
Prod checklist
- Both indexes built from the same enriched chunk text, from the chunk registry
- One filter-builder used by both legs; parity asserted in tests
- Analyzer preserves identifiers; analyzer test suite in CI
- Parallel retrieval, RRF fusion (k=60), pool ~50 → cross-encoder rerank → top 5–8
- Per-index chunk-count monitoring with divergence alerts
- Match-source telemetry (dense-only / sparse-only / both) on cited chunks
- Golden-set recall reported for dense-only, sparse-only, hybrid, split by query type
- Both indexes included in migration plans when indexed text changes (versioning & migration)
Metrics to watch
| Metric | Why |
|---|---|
| recall@50: dense / sparse / hybrid, by query type | Justifies hybrid; finds regressions |
| Sparse-only contribution to cited chunks | How much BM25 is earning |
| Index chunk-count divergence | Sync bugs |
| Analyzer test suite pass rate | Identifier matching |
Related
- Multi-query + RRF — same fusion machinery
- Reranking — required after fusion
- Filtered search — filter parity
- Contextual retrieval — enriched text helps BM25 too
- Abstention — RRF's loss of score magnitude complicates thresholds