Query understanding → HyDE (Hypothetical Document Embeddings)
What it is
Dense retrieval matches a query vector against passage vectors. But "how long is the notice period?" (7 words) and the actual clause (200 words of legal prose) are stylistically very different, even when semantically related. HyDE (Gao et al., 2022) closes the gap by generating a fake document that answers the query, then embedding the fake document instead of (or alongside) the query:
query: "how much notice to terminate?"
│ LLM (no retrieval, may hallucinate — that's fine)
▼
hypothetical: "Either party may terminate this Agreement upon thirty (30) days' prior
written notice to the other party. Notice must be delivered..."
│ embed as a DOCUMENT (passage-side prefix)
▼
search → real clauses that look like this
The hallucinated specifics (30 days) don't matter — the embedding captures the shape and vocabulary of the answer, and the nearest real passages are the ones you want. Accuracy comes from retrieval and grounding downstream, never from the hypothetical.
How to implement
- Generate with the fast tier, temperature ~0.3–0.7,
max_tokens~150–250: "Write a short passage that would answer this question, in the style of the source documents. Do not hedge; just write the passage." - Optionally generate n=2–3 hypotheticals and average their embeddings (reduces variance from a single unlucky generation).
- Embed with the document/passage prefix (symmetric vs asymmetric) — this is the whole point; it's document-like text.
- Search with that vector. Common production form: fuse the HyDE vector's results with the original query's dense results and BM25 results via RRF (multi-query + Reciprocal Rank Fusion) rather than replacing them.
- Never show the hypothetical to the user and never let it enter the generation context — it's a hallucination by construction.
- Gate it: only for short/underspecified queries, or when the original query's top score is below a floor (score thresholding & abstention).
- Evaluate: recall@k with HyDE vs without on the golden set, split by query length.
Small example
HYDE_SYS = """Write a single short passage (3-5 sentences) that would plausibly appear in an internal
company document and would answer the user's question. Write it in the style of a policy or contract
excerpt. State specifics confidently even if you must invent them. Output the passage only."""
def hyde_vector(query: str, n: int = 2):
docs = []
for _ in range(n):
r = llm("hyde", HYDE_SYS, [{"role": "user", "content": query}], max_tokens=220, temperature=0.5)
docs.append(r.text.strip())
vecs = embed_documents(docs) # DOCUMENT side prefix — critical (symmetric vs asymmetric)
v = vecs.mean(axis=0)
return v / np.linalg.norm(v), docs
def retrieve_with_hyde(query, caller, k=25):
flt = build_acl_filter(caller)
qv = embed_queries([query])[0]
hv, hypotheticals = hyde_vector(query)
log_hyde(query, hypotheticals) # logged for debugging, never shown, never in context
lists = [
[h.id for h in vec.search(qv, k, flt)], # original query, query-side embedding
[h.id for h in vec.search(hv, k, flt)], # hypothetical answer, doc-side embedding
[h.id for h in bm25.search(query, k, flt)], # keywords
]
fused = rrf(lists)[:50]
return reranker.rerank(query, [chunk_store.get(d) for d, _ in fused])[:8]
SHORT = lambda q: len(q.split()) <= 8
def should_hyde(query, top_score): return SHORT(query) or top_score < 0.35
Pros
- Closes the query/document style gap — the main win, especially for short queries against verbose corpora (legal, policy, scientific).
- No training, no index change — purely a query-side technique.
- Composes with hybrid and multi-query via RRF.
- Helps most exactly where dense retrieval is weakest: terse, underspecified questions.
Cons
- An LLM call on the critical path (~200–500 ms) plus tokens.
- Hallucinated hypotheticals can mislead retrieval into a wrong topic — particularly for questions about entities the model has wrong priors about ("what is our SKU-4471 return policy" → invents a consumer-electronics policy).
- No help when the corpus doesn't contain the answer — it can actively make abstention harder by producing confident-looking retrieval.
- Wrong-side embedding (using the query prefix on the hypothetical) silently negates the benefit.
- Adds variance: two runs can retrieve differently unless temperature is low or n>1 with averaging.
When to use
- Short, terse, or keyword-poor queries against verbose documents.
- Domains with large query/document vocabulary gaps (legal, medical, standards).
- As one list among several in an RRF fusion, gated by a low top-score signal.
When NOT to use
- Long, well-specified queries — nothing to close; pure cost.
- Highly entity-specific lookups ("status of ticket 4471") — BM25 does this better and HyDE invents noise.
- Latency-critical paths.
- As a replacement for the original query vector; keep both.
Production at scale
Gate hard. Run HyDE only for queries that fail a cheap test: short length, or the original dense search's top score below a floor. Typical production gating fires on 10–30% of traffic and captures most of the benefit at a fraction of the cost.
Two-pass architecture. Search with the original query first (it's already needed); if the top score clears the floor, answer. Otherwise, generate the hypothetical and search again. This makes HyDE a recovery step rather than a fixed cost, and it composes with the escalation ladder (routing & escalation).
Document-side embedding, enforced. Use embed_documents() for the hypothetical; a unit test should assert the HyDE path calls the document embedder. This is the single most common HyDE bug and it fails silently (symmetric vs asymmetric).
Cache hypotheticals by normalised query with a medium TTL; repeated queries are common in support and internal search.
Never in context. The hypothetical must not reach the answer prompt, the citations, or the logs shown to users. Store it in the trace for debugging only, marked clearly as synthetic (observability & tracing). A hypothetical leaking into generation is a hallucination injected by your own pipeline (failure modes: poisoning, distraction, confusion, clash poisoning).
Domain steering. Give the generator a one-line description of the corpus ("internal HR policies for a German employer") so hypotheticals match your documents' register, not generic web text. Tenant-specific overlay if needed.
Evaluation split. Report recall by query length bucket; HyDE's gain is concentrated in the short bucket, and a global average can hide both the gain and a regression on long queries.
Cost model. One fast-tier call (~250 output tokens) per gated query. At 20% gating and 100k queries/day that's 20k calls — small, but track it as its own line item next to multi-query (multi-query + Reciprocal Rank Fusion), because the two together can double query-path LLM spend.
Prod checklist
- Gated by query length and/or low top-score from a first-pass search
- Fast tier, bounded
max_tokens, cached system prompt with corpus-register description - Embedded with the document prefix; unit test asserts it
- Fused via RRF with original-query dense + BM25 lists; reranked with the original query
- Hypothetical never enters generation context or user-visible logs; stored in trace as synthetic
- Hypothetical cache by normalised query
- Golden-set recall reported by query-length bucket
- LLM cost tracked as a distinct query-path line item
Metrics to watch
| Metric | Why |
|---|---|
| recall@k with vs without HyDE, by query length | Where the value is |
| Gating rate | Cost discipline |
| Added p95 latency on gated queries | UX |
| Abstention rate change | HyDE can mask "no answer exists" |
Related
- Symmetric vs asymmetric — the document-side prefix is mandatory here
- Multi-query + RRF — HyDE is one more list to fuse
- Score thresholding — the gating signal and the abstention risk
- Context poisoning — why the hypothetical must never reach the context