Retrieval & ranking → Reranking (cross-encoder / late interaction)

Reranking in RAG Explained: Cross-Encoder Rerank, Keep the Best Five

What it is

Retrieval and ranking are different problems:

  • Bi-encoder (retrieval): encodes query and passage separately. The passage vector was computed at ingest without knowledge of the query. Fast — you can search millions — but coarse.
  • Cross-encoder (reranking): takes (query, passage) as one input and outputs a relevance score, with full attention across both. Vastly more accurate; far too slow to run over the corpus, but fine over 50 candidates.
  • Late interaction (ColBERT-style): stores per-token vectors and computes MaxSim between query and passage tokens. Middle ground — better than bi-encoders, cheaper than cross-encoders, but a much larger index.
query ──► hybrid retrieve (k=50, ~20 ms) ──► cross-encoder scores 50 pairs (~50-200 ms)
                                          ──► top 5-8 ──► context assembly ──► LLM

Typical gains: nDCG@5 improvements of 10–30% over the retrieval-only ranking, and — more importantly for RAG — the right passage moves into position 1–3 where the model actually uses it ("Lost in the middle").

Options: hosted APIs (Cohere Rerank, Voyage rerank, Jina reranker), open-source cross-encoders (BGE-reranker-v2-m3, mxbai-rerank, MiniLM cross-encoders), ColBERT/ColPali for late interaction, or an LLM-as-reranker (a fast model scoring relevance) when nothing else is available.

How to implement

  1. Retrieve a wider pool than you need — k=30–100 from hybrid fusion (hybrid search). The reranker can only fix ordering, never recover a passage retrieval missed. Pool size is a recall decision.
  2. Score pairs: send (query, passage_text) for all candidates in one batched call. Use the original user question (not a HyDE hypothetical or a paraphrase, multi-query + Reciprocal Rank Fusion/04.4.3).
  3. Truncate passages to the reranker's max length (often 512 tokens); for parent–child, rerank the children and then collapse to parents (parent–child) — cross-encoders degrade on long inputs.
  4. Cut to top-n (5–8 typical) or apply a score threshold (score thresholding & abstention) — the reranker's calibrated score is a much better abstention signal than fused RRF ranks.
  5. Fall back gracefully: on reranker timeout or error, use the fusion order and log it (failure handling). Never fail the request because the reranker is down.
  6. Cache by (query_hash, chunk_id) with a short TTL for repeated queries.
  7. Evaluate: nDCG@5 and position-of-first-relevant, before and after; and end-to-end answer quality, which is what actually pays.

Small example

# --- Hosted reranker (Cohere-style) with timeout + fallback ---
def rerank_hosted(query: str, candidates: list[dict], top_n: int = 6, timeout_s: float = 1.5):
    docs = [c["text"][:2000] for c in candidates]
    try:
        r = cohere_client.rerank(model="rerank-v3.5", query=query, documents=docs,
                                 top_n=top_n, request_options={"timeout_in_seconds": timeout_s})
        out = [candidates[x.index] | {"rerank_score": x.relevance_score} for x in r.results]
        log_rerank(query, n_in=len(candidates), n_out=len(out), top_score=out[0]["rerank_score"] if out else 0.0)
        return out
    except Exception as e:
        log_rerank_fallback(query, str(e))
        return candidates[:top_n]                      # degrade to fusion order, never fail the request

# --- Self-hosted cross-encoder ---
from sentence_transformers import CrossEncoder
ce = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)     # load once per worker

def rerank_local(query: str, candidates: list[dict], top_n: int = 6, batch_size: int = 32):
    pairs = [(query, c["text"]) for c in candidates]
    scores = ce.predict(pairs, batch_size=batch_size, show_progress_bar=False)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c | {"rerank_score": float(s)} for c, s in ranked[:top_n]]

# --- Parent-child: rerank children, then collapse ---
def rerank_then_collapse(query, child_candidates, n_parents=4):
    scored = rerank_local(query, child_candidates, top_n=len(child_candidates))
    best = {}
    for c in scored:
        pid = c["parent_id"]
        if pid not in best or c["rerank_score"] > best[pid]["rerank_score"]:
            best[pid] = c
    top = sorted(best.values(), key=lambda c: -c["rerank_score"])[:n_parents]
    return [parent_store.get(c["parent_id"]) | {"rerank_score": c["rerank_score"], "matched_child": c["chunk_id"]} for c in top]

Pros

  • The largest single quality jump in most RAG pipelines — usually bigger than changing embedding models or LLMs.
  • Lets you retrieve wide and cut narrow: better recall from a big pool, better precision in the final context, fewer tokens to the LLM.
  • Calibrated relevance scores enable thresholding and abstention (score thresholding & abstention) far better than fused ranks.
  • No re-indexing required — a query-time component you can swap, tune, or fine-tune independently (fine-tuning embeddings on your own logs).
  • Fine-tuning a reranker on your logs is cheaper than fine-tuning an embedder and needs no migration.

Cons

  • Latency: 50–300 ms depending on pool size, model, and hosting. It sits on the critical path.
  • Cost: hosted rerankers charge per search/document; self-hosting needs GPU capacity.
  • Max length limits mean long chunks get truncated — rerank children, not 2,000-token parents.
  • Cannot fix retrieval misses — garbage pool in, garbage top-5 out.
  • Another dependency that can time out; needs a fallback path.
  • Multilingual and domain performance varies; validate on your data.

When to use

Essentially always, once you have a working retrieval baseline. If you can add only one component to a naive RAG pipeline, add this one.

When NOT to use

  • Ultra-low-latency paths (sub-100 ms end-to-end budgets) where 100 ms of reranking is unaffordable — consider a smaller model, a smaller pool, or ColBERT-style late interaction.
  • Tiny corpora where k=5 retrieval already returns everything relevant.
  • When the pool is already perfect (measure: if the retrieval order and reranked order agree on the top-5 nearly always, the reranker is only costing you latency — rare in practice).

Production at scale

Pool size is the recall/latency dial. Reranking cost is linear in pool size. Sweep k ∈ {20, 50, 100} on the golden set and plot end-to-end quality vs p95 latency; most systems plateau around 50. Under load, shed pool size before shedding the reranker entirely (capacity & limits).

Self-host vs API. At high QPS, a self-hosted BGE-reranker on a GPU is far cheaper than per-document API pricing, and gives you latency control and residency. At low volume, the API is simpler and often better quality. Model the cross-over.

Batch and warm. One batched call for all candidates, not one call per candidate. Keep the model loaded per worker; pre-warm replicas before routing traffic.

Fine-tune it. The same (query, positive, hard-negative) triplets used for embedding fine-tuning (fine-tuning embeddings on your own logs) train a cross-encoder — and the gain is usually larger, with no re-index. For multi-tenant systems this is the preferred adaptation path: you can even serve per-tenant reranker adapters.

Timeout and fallback are mandatory. Set an aggressive timeout (1–2 s), fall back to fusion order, log every fallback, and alert on the rate. A reranker outage should degrade quality, not availability (failure handling).

Score calibration. Reranker scores are model-specific; calibrate a threshold on the golden set for abstention (score thresholding & abstention) and re-calibrate on every model change. Log the top score on every query — it's your best single relevance telemetry.

Two-stage reranking at very large pools: a fast, cheap reranker (or ColBERT) on 200 candidates → a strong cross-encoder on the surviving 30. Worth it only when recall genuinely requires a 200-deep pool.

Watch the truncation. If passages exceed the reranker's max length, you're scoring prefixes. Rerank children (200–300 tokens) and collapse to parents afterwards — this is the standard production shape (parent–child).

Prod checklist

  • Pool of 30–100 from hybrid fusion; reranked with the original query
  • Children reranked, then collapsed to parents; passages within max length
  • Batched inference; model warm per worker (self-hosted) or timeout-bounded (API)
  • Timeout + fallback to fusion order; fallback rate alerted
  • Score threshold calibrated on golden set; top score logged per query
  • Pool-size sweep documented (quality vs p95); load-shedding reduces pool first
  • Fine-tuning pipeline from production triplets
  • Cache by (query_hash, chunk_id) with short TTL

Metrics to watch

MetricWhy
nDCG@5 and position-of-first-relevant, before/after rerankIsolates the rerank gain
Rerank p95 latency; fallback rateReliability
Top rerank score distributionCalibration and abstention
Answer quality vs pool sizeRight k