HyDE Explained: Hypothetical Document Embeddings for RAG
HyDE improves RAG retrieval by having an LLM write a fake answer first, then searching with that answer's embedding instead of the raw query.
HyDE, short for Hypothetical Document Embeddings, is a retrieval technique that asks a language model to write a plausible answer to a question before searching, then uses the embedding of that made-up answer — not the original question — to find real documents. The hypothetical answer may contain wrong details, but it looks like the kind of passage you want to retrieve, and that resemblance is what makes the search work better.
The problem HyDE solves
In a typical retrieval-augmented generation pipeline, a user’s query is converted into a vector and compared against vectors for every chunk in your knowledge base. The closest chunks get stuffed into the prompt.
The weak point is that queries and documents are different kinds of text. A query is short, vague, and phrased as a question: “why does my container keep getting OOM killed?” The document that answers it is long, declarative, and full of domain vocabulary: “When a process inside a cgroup exceeds its memory limit, the kernel’s out-of-memory killer terminates it…”
Vector embeddings capture meaning, but they also capture form. A short question and a long explanatory paragraph can land in noticeably different regions of embedding space even when one perfectly answers the other. This is sometimes called the query-document asymmetry problem, and it is one reason retrieval quality often disappoints on real user questions.
How HyDE works
HyDE sidesteps the asymmetry by converting the query into a document first. The pipeline has four steps:
- Generate. Send the user’s query to an LLM with an instruction like “Write a short passage that answers this question.” The model produces a hypothetical document.
- Embed. Run the hypothetical document through the same embedding model you used to index your corpus.
- Search. Use that vector for nearest-neighbor search against your index, exactly as you would with a normal query vector.
- Answer. Pass the real retrieved documents to the LLM to generate the final, grounded response.
The hypothetical document is thrown away after step 3. It never reaches the user and is never treated as a source of truth. Its only job is to be a better search key.
Some implementations generate several hypothetical documents and average their embeddings, which smooths out the quirks of any single generation. Others include the original query’s embedding in that average so the user’s exact wording still carries weight.
Why a wrong answer can still retrieve the right documents
The counterintuitive part of HyDE is that the generated passage can be factually wrong and the technique still helps. That works because the embedding model cares more about topic, vocabulary, and structure than about whether specific claims are true.
If you ask “what flag disables the Postgres autovacuum for a single table?”, the LLM might invent an incorrect flag name. But its hypothetical answer will still talk about tables, storage parameters, autovacuum, and ALTER TABLE — which is exactly the neighborhood where the correct documentation chunk lives. The embedding acts as a fuzzy pointer to the right region of the corpus, and the real documents supply the facts.
This is also why HyDE does not increase hallucination in the final answer, provided you build the pipeline correctly. The final generation step sees only retrieved text, not the hypothetical draft.
HyDE vs standard query embedding
| Standard query embedding | HyDE | |
|---|---|---|
| Search key | Embedding of the raw query | Embedding of an LLM-written answer |
| Extra LLM calls | None | One (or more) per query |
| Latency | Lowest | Higher — adds a generation step |
| Cost | Embedding only | Embedding plus generation tokens |
| Handles vague or short queries | Poorly | Well |
| Needs labeled training data | No | No |
| Risk | Misses relevant docs phrased differently | Can drift if the LLM misreads the query |
HyDE was originally proposed as a zero-shot method: it improves dense retrieval without any relevance labels or fine-tuning of the embedding model. That makes it attractive when you are building on a new corpus and have no query-document pairs to train on.
When HyDE helps most
HyDE tends to pay off in a few specific situations:
- Short or ambiguous queries. A three-word query has little signal; a hundred-word hypothetical passage has much more.
- Specialist corpora. Legal, medical, and internal engineering documents use jargon users rarely type. The LLM translates casual phrasing into domain vocabulary.
- Question-answer mismatch. FAQ-style questions against prose documentation, or natural-language questions against reference material.
- No training data. If you cannot fine-tune your embedding model, HyDE is one of the cheapest ways to improve recall.
When to skip it
HyDE is not free and not universally better.
- Latency-sensitive paths. An extra generation call adds noticeable time before retrieval can even start. For autocomplete or real-time chat, that may be unacceptable.
- Exact-match queries. Searching for an error code, a SKU, or a function name works better with keyword search. A hypothetical paragraph dilutes the one token that matters. This is where hybrid search combining BM25 and vectors shines.
- Topics the LLM knows nothing about. If your corpus covers proprietary systems the model has never seen, its hypothetical answer may point to the wrong neighborhood entirely.
- Cost at scale. Every query now consumes generation tokens. Estimate the impact with an LLM token cost calculator before rolling it out to all traffic.
Combining HyDE with other retrieval techniques
HyDE slots in cleanly alongside other parts of a modern retrieval stack. A common pattern is:
- Run HyDE-based vector search and keyword search in parallel.
- Merge the candidate lists.
- Use a reranker — a cross-encoder that scores each candidate against the original query — to choose the final top results.
The reranker step is important. HyDE is good at casting a wide, semantically relevant net, while the reranker is good at precision. Scoring against the original query keeps the final selection anchored to what the user actually asked.
You can also gate HyDE behind a simple heuristic: use it only when the query is short, or only when the first-pass retrieval scores fall below a confidence threshold. That keeps latency low for easy queries and spends the extra LLM call only where it is likely to help.
Implementation tips
- Keep the generation prompt short and neutral. Ask for “a passage that would answer this question” rather than a full chat-style response. You want document-shaped text.
- Match the style of your corpus. If your documents are API references, ask for reference-style output. The closer the hypothetical document is to real ones, the better the match.
- Use a small, fast model. The hypothetical document does not need to be correct, so a cheaper model often works as well as a large one.
- Evaluate recall, not vibes. Build a small test set of queries with known relevant documents and compare recall at k with and without HyDE before committing.
The takeaway
HyDE improves retrieval by searching with a fake answer instead of the question. An LLM drafts a hypothetical document, that draft is embedded and used as the search key, and only real retrieved documents reach the final generation step. It shines on short, vague, or jargon-heavy queries and requires no labeled data, but it adds latency and cost and does little for exact-match lookups. Treat it as one tool in the retrieval stack — pair it with keyword search and a reranker, and measure recall before and after.
Tagged
Keep reading
Chisato · · 5 min read Prefill vs Decode: The Two Phases of LLM Inference
LLM inference runs in two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Why they behave so differently.
Chisato · · 5 min read LLM Perplexity Explained: What the Metric Measures
Perplexity measures how surprised a language model is by text: the exponent of its average per-token loss. How it is computed, read, and misread.
Chisato · · 5 min read LLM Knowledge Cutoff Explained: What Models Don't Know
An LLM's knowledge cutoff is the point after which its training data stops. Why it exists, how it causes stale answers, and how to work around it.