Retrieval & ranking → Reranking (cross-encoder / late interaction)
What it is
Retrieval and ranking are different problems:
- Bi-encoder (retrieval): encodes query and passage separately. The passage vector was computed at ingest without knowledge of the query. Fast — you can search millions — but coarse.
- Cross-encoder (reranking): takes
(query, passage)as one input and outputs a relevance score, with full attention across both. Vastly more accurate; far too slow to run over the corpus, but fine over 50 candidates. - Late interaction (ColBERT-style): stores per-token vectors and computes MaxSim between query and passage tokens. Middle ground — better than bi-encoders, cheaper than cross-encoders, but a much larger index.
query ──► hybrid retrieve (k=50, ~20 ms) ──► cross-encoder scores 50 pairs (~50-200 ms)
──► top 5-8 ──► context assembly ──► LLM
Typical gains: nDCG@5 improvements of 10–30% over the retrieval-only ranking, and — more importantly for RAG — the right passage moves into position 1–3 where the model actually uses it ("Lost in the middle").
Options: hosted APIs (Cohere Rerank, Voyage rerank, Jina reranker), open-source cross-encoders (BGE-reranker-v2-m3, mxbai-rerank, MiniLM cross-encoders), ColBERT/ColPali for late interaction, or an LLM-as-reranker (a fast model scoring relevance) when nothing else is available.
How to implement
- Retrieve a wider pool than you need — k=30–100 from hybrid fusion (hybrid search). The reranker can only fix ordering, never recover a passage retrieval missed. Pool size is a recall decision.
- Score pairs: send
(query, passage_text)for all candidates in one batched call. Use the original user question (not a HyDE hypothetical or a paraphrase, multi-query + Reciprocal Rank Fusion/04.4.3). - Truncate passages to the reranker's max length (often 512 tokens); for parent–child, rerank the children and then collapse to parents (parent–child) — cross-encoders degrade on long inputs.
- Cut to top-n (5–8 typical) or apply a score threshold (score thresholding & abstention) — the reranker's calibrated score is a much better abstention signal than fused RRF ranks.
- Fall back gracefully: on reranker timeout or error, use the fusion order and log it (failure handling). Never fail the request because the reranker is down.
- Cache by
(query_hash, chunk_id)with a short TTL for repeated queries. - Evaluate: nDCG@5 and position-of-first-relevant, before and after; and end-to-end answer quality, which is what actually pays.
Small example
# --- Hosted reranker (Cohere-style) with timeout + fallback ---
def rerank_hosted(query: str, candidates: list[dict], top_n: int = 6, timeout_s: float = 1.5):
docs = [c["text"][:2000] for c in candidates]
try:
r = cohere_client.rerank(model="rerank-v3.5", query=query, documents=docs,
top_n=top_n, request_options={"timeout_in_seconds": timeout_s})
out = [candidates[x.index] | {"rerank_score": x.relevance_score} for x in r.results]
log_rerank(query, n_in=len(candidates), n_out=len(out), top_score=out[0]["rerank_score"] if out else 0.0)
return out
except Exception as e:
log_rerank_fallback(query, str(e))
return candidates[:top_n] # degrade to fusion order, never fail the request
# --- Self-hosted cross-encoder ---
from sentence_transformers import CrossEncoder
ce = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512) # load once per worker
def rerank_local(query: str, candidates: list[dict], top_n: int = 6, batch_size: int = 32):
pairs = [(query, c["text"]) for c in candidates]
scores = ce.predict(pairs, batch_size=batch_size, show_progress_bar=False)
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
return [c | {"rerank_score": float(s)} for c, s in ranked[:top_n]]
# --- Parent-child: rerank children, then collapse ---
def rerank_then_collapse(query, child_candidates, n_parents=4):
scored = rerank_local(query, child_candidates, top_n=len(child_candidates))
best = {}
for c in scored:
pid = c["parent_id"]
if pid not in best or c["rerank_score"] > best[pid]["rerank_score"]:
best[pid] = c
top = sorted(best.values(), key=lambda c: -c["rerank_score"])[:n_parents]
return [parent_store.get(c["parent_id"]) | {"rerank_score": c["rerank_score"], "matched_child": c["chunk_id"]} for c in top]
Pros
- The largest single quality jump in most RAG pipelines — usually bigger than changing embedding models or LLMs.
- Lets you retrieve wide and cut narrow: better recall from a big pool, better precision in the final context, fewer tokens to the LLM.
- Calibrated relevance scores enable thresholding and abstention (score thresholding & abstention) far better than fused ranks.
- No re-indexing required — a query-time component you can swap, tune, or fine-tune independently (fine-tuning embeddings on your own logs).
- Fine-tuning a reranker on your logs is cheaper than fine-tuning an embedder and needs no migration.
Cons
- Latency: 50–300 ms depending on pool size, model, and hosting. It sits on the critical path.
- Cost: hosted rerankers charge per search/document; self-hosting needs GPU capacity.
- Max length limits mean long chunks get truncated — rerank children, not 2,000-token parents.
- Cannot fix retrieval misses — garbage pool in, garbage top-5 out.
- Another dependency that can time out; needs a fallback path.
- Multilingual and domain performance varies; validate on your data.
When to use
Essentially always, once you have a working retrieval baseline. If you can add only one component to a naive RAG pipeline, add this one.
When NOT to use
- Ultra-low-latency paths (sub-100 ms end-to-end budgets) where 100 ms of reranking is unaffordable — consider a smaller model, a smaller pool, or ColBERT-style late interaction.
- Tiny corpora where k=5 retrieval already returns everything relevant.
- When the pool is already perfect (measure: if the retrieval order and reranked order agree on the top-5 nearly always, the reranker is only costing you latency — rare in practice).
Production at scale
Pool size is the recall/latency dial. Reranking cost is linear in pool size. Sweep k ∈ {20, 50, 100} on the golden set and plot end-to-end quality vs p95 latency; most systems plateau around 50. Under load, shed pool size before shedding the reranker entirely (capacity & limits).
Self-host vs API. At high QPS, a self-hosted BGE-reranker on a GPU is far cheaper than per-document API pricing, and gives you latency control and residency. At low volume, the API is simpler and often better quality. Model the cross-over.
Batch and warm. One batched call for all candidates, not one call per candidate. Keep the model loaded per worker; pre-warm replicas before routing traffic.
Fine-tune it. The same (query, positive, hard-negative) triplets used for embedding fine-tuning (fine-tuning embeddings on your own logs) train a cross-encoder — and the gain is usually larger, with no re-index. For multi-tenant systems this is the preferred adaptation path: you can even serve per-tenant reranker adapters.
Timeout and fallback are mandatory. Set an aggressive timeout (1–2 s), fall back to fusion order, log every fallback, and alert on the rate. A reranker outage should degrade quality, not availability (failure handling).
Score calibration. Reranker scores are model-specific; calibrate a threshold on the golden set for abstention (score thresholding & abstention) and re-calibrate on every model change. Log the top score on every query — it's your best single relevance telemetry.
Two-stage reranking at very large pools: a fast, cheap reranker (or ColBERT) on 200 candidates → a strong cross-encoder on the surviving 30. Worth it only when recall genuinely requires a 200-deep pool.
Watch the truncation. If passages exceed the reranker's max length, you're scoring prefixes. Rerank children (200–300 tokens) and collapse to parents afterwards — this is the standard production shape (parent–child).
Prod checklist
- Pool of 30–100 from hybrid fusion; reranked with the original query
- Children reranked, then collapsed to parents; passages within max length
- Batched inference; model warm per worker (self-hosted) or timeout-bounded (API)
- Timeout + fallback to fusion order; fallback rate alerted
- Score threshold calibrated on golden set; top score logged per query
- Pool-size sweep documented (quality vs p95); load-shedding reduces pool first
- Fine-tuning pipeline from production triplets
- Cache by (query_hash, chunk_id) with short TTL
Metrics to watch
| Metric | Why |
|---|---|
| nDCG@5 and position-of-first-relevant, before/after rerank | Isolates the rerank gain |
| Rerank p95 latency; fallback rate | Reliability |
| Top rerank score distribution | Calibration and abstention |
| Answer quality vs pool size | Right k |
Related
- Hybrid search — supplies the pool
- Parent–child Parent–child — rerank children, return parents
- Score thresholding — uses reranker scores
- Fine-tuning — applies to rerankers, with no re-index
- Lost in the middle — reranking is what makes ordering meaningful