Building RAG From Scratch, Part 3: Hybrid Retrieval With Reciprocal Rank Fusion
Series: Building RAG From Scratch. Part 3 of ~6. Part 1 built BM25, Part 2 built embeddings, and both have now failed in specific, demonstrable ways on the same corpus. This part combines them.
Part 1 and Part 2 each tell half the story. BM25 nails exact codes and fails on paraphrase (“car” vs “automobile”). Embeddings fix paraphrase and can route a bare ID to the wrong document when two systems share digits. Neither is strictly better — they fail on different inputs. The obvious next move is to run both and combine the results.
The obvious wrong way to combine them is averaging the raw scores.
Why you can’t just average the scores
Look at the actual numbers from Part 1 and Part 2 side by side, same corpus, same query:
| doc | BM25 | Embeddings |
|---|---|---|
| doc0 (query: “car oil change”) | 2.4034 | 0.5903 |
| doc4 (same query) | 2.4034 | 0.6441 |
BM25 scores here range roughly 0 to 4.5 depending on term rarity and query length, unbounded above. Embedding cosine similarity is mathematically bounded in [-1, 1]. If you average these directly, BM25 dominates every single query just because its numbers are bigger — not because it’s more relevant, just because of unit mismatch. You could normalize both to [0, 1] first, but then you’re picking an arbitrary normalization scheme (min-max over what set of documents? per-query or per-corpus?) that has no principled justification and silently breaks the moment either retriever’s score distribution shifts — a longer query, a different embedding model, a bigger corpus, and your hand-tuned weights are wrong again.
Reciprocal Rank Fusion: fuse ranks, not scores
RRF sidesteps the scale problem by throwing away the raw scores entirely and working with rank position instead. Each retriever independently produces a ranked list; a document’s fused score is the sum of 1 / (k + rank) across every list it appears in:
RRF(d) = Σ over each retriever r: 1 / (k + rank_r(d))
rank_r(d) is the document’s 1-indexed position in retriever r’s ranked list (1 = top result). k is a smoothing constant — commonly 60 — that controls how steeply the fusion favors top-ranked results over lower ones. A document ranked #1 by a retriever contributes 1/(k+1); ranked #10, it contributes 1/(k+10), a much smaller amount. A document that doesn’t appear in a retriever’s result list at all simply contributes nothing from that retriever — no penalty beyond “doesn’t help.”
from collections import defaultdict
def reciprocal_rank_fusion(ranked_lists: list[list[int]], k: int = 60) -> list[tuple[int, float]]:
fused_scores: dict[int, float] = defaultdict(float)
for ranked_list in ranked_lists:
for rank, doc_idx in enumerate(ranked_list): # 0-indexed
fused_scores[doc_idx] += 1 / (k + rank + 1) # +1 -> rank 1 is top
return sorted(fused_scores.items(), key=lambda pair: pair[1], reverse=True)
class HybridRetriever:
def __init__(self, retrievers: list, k: int = 60):
self.retrievers = retrievers
self.k = k
def search(self, query: str, top_k: int = 5, candidate_k: int = 10) -> list[tuple[int, float]]:
ranked_lists = [
[doc_idx for doc_idx, _score in retriever.search(query, top_k=candidate_k)]
for retriever in self.retrievers
]
return reciprocal_rank_fusion(ranked_lists, k=self.k)[:top_k]
No score normalization, no weighting scheme to tune. Both retrievers vote by rank, and RRF counts the votes.
Running it on the original queries
Here’s BM25, embeddings, and the fused RRF score together on the five queries from Part 1 and Part 2:
Query: "car oil change"
| doc | BM25 | Embed | RRF |
|---|---|---|---|
| doc0: car/oil/brakes, natural | 2.4034 | 0.5903 | 0.032522 |
| doc4: car/oil, overlaps doc0 | 2.4034 | 0.6441 | 0.032522 |
| doc5: “oil” × 11, spam | 1.3915 | 0.3865 | 0.031498 |
Query: "automobile brake service"
| doc | BM25 | Embed | RRF |
|---|---|---|---|
| doc1: automobile/oil/brakes | 4.1587 | 0.5647 | 0.032787 |
| doc0: car/oil/brakes, natural | 1.1432 | 0.4842 | 0.032258 |
Both retrievers already agreed doc1 was the answer here, so fusion just confirms it with the highest RRF score in the set (0.032787 — the maximum possible when a document is rank 1 in every retriever: 2 × 1/(60+1) = 0.032787).
Query: "SKU-48213-B"
| doc | BM25 | Embed | RRF |
|---|---|---|---|
| doc2: exact SKU code | 1.7104 | 0.5710 | 0.032787 |
Same story — both retrievers independently nail this, fusion just reflects it.
Where it gets interesting: ties
Query: "weekend weather"
| doc | BM25 | Embed | RRF |
|---|---|---|---|
| doc4: car dealership, says “weekend” | 1.7104 (rank 1) | 0.3379 (rank 2) | 0.032522 |
| doc3: actually about weather | 1.5868 (rank 2) | 0.3948 (rank 1) | 0.032522 |
Part 1 flagged this as lexical retrieval’s worst failure (doc4 beats doc3 on BM25, for the wrong reason). Part 2 showed embeddings fix it decisively (doc3 beats doc4, 0.3948 to 0.3379 — not close). You’d expect the hybrid to clearly side with embeddings here. It doesn’t: both documents get the exact same RRF score, 0.032522.
This isn’t rounding. Work out the math: doc4 is rank 1 in BM25 and rank 2 in embeddings, so its RRF score is 1/(60+1) + 1/(60+2) = 0.032522. doc3 is rank 2 in BM25 and rank 1 in embeddings — 1/(60+2) + 1/(60+1), the identical sum, just the two terms swapped. Whenever two documents swap which one ranks #1 vs #2 between two equally-weighted retrievers, RRF always ties them exactly, no matter how decisive either retriever’s actual preference was. BM25’s lead for doc4 was a coin flip (1.7104 vs 1.5868); embeddings’ lead for doc3 was decisive (0.3948 vs 0.3379, about 17% ahead). RRF can’t see that difference — it only sees “rank 1” and “rank 2,” and a rank is a rank regardless of the score gap behind it.
A sharper version of the same limitation
To see this isn’t a one-off, I constructed a case specifically to probe it: a query mixing a semantic signal with an exact code string that belongs to the wrong document.
Query: "automobile SKU-48213-B service"
| doc | BM25 | Embed | RRF |
|---|---|---|---|
| doc1: automobile/oil/brakes (correct answer) | 2.9692 (rank 1) | 0.3644 (rank 2) | 0.032522 |
| doc2: exact SKU code (wrong answer here) | 1.7104 (rank 2) | 0.4920 (rank 1) | 0.032522 |
Embeddings alone get this wrong: the exact code string pulls doc2 to the top (0.4920 vs 0.3644), even though doc1 — matching “automobile” and “service” — is clearly the intended document. BM25 alone gets it right, and decisively (2.9692 vs 1.7104, doc1 almost double doc2’s score). You’d hope fusion inherits BM25’s confidence and corrects embeddings’ mistake outright.
Instead: another exact tie. Same mechanism as “weekend weather” — doc1 is rank 1 in BM25 / rank 2 in embeddings, doc2 is the mirror image, so the sums are identical. The hybrid does make real progress here (doc2’s wrongful outright lead collapses from a clear win to a tie), but it doesn’t produce a clean correct answer. A tie-breaking rule (e.g. prefer the retriever with the larger score gap, or weight BM25 slightly higher when it has an exact token match) would resolve this in favor of doc1 — but that’s an extra design decision on top of RRF, not something RRF gives you by default.
What this means in practice
- RRF is a strong default precisely because it needs no score normalization or hand-tuned weights — just rank the same corpus with each retriever and sum reciprocal ranks. That’s why it’s a common first choice for combining lexical and semantic search in production systems.
- It discards magnitude on purpose, which is also its blind spot. Two retrievers that are “barely in disagreement” (BM25’s near-tie on “weekend weather”) and two that are “sharply in disagreement” (BM25’s near-2x margin on the SKU query) get fused identically if the rank positions are the same. If you need the fusion to reflect how confident each retriever was, not just which document it preferred, RRF alone isn’t enough — you’d want a learned fusion (weight each retriever’s contribution based on historical reliability) or a reranking stage that looks at the actual query-document pairs again, with more signal than a rank number.
- This is exactly what Part 4 (reranking) adds. Hybrid retrieval here is a recall stage — cast a wide net combining two different notions of relevance. A cross-encoder reranker in the next part looks at each retrieved candidate against the query directly (not just independently-computed scores) and can break exactly the kind of tie RRF can’t: “automobile SKU-48213-B service” against doc1 vs doc2, scored jointly, not fused from two separate opinions.
What’s next
Part 4 takes the fused candidates from this part’s hybrid retriever and reranks them with a cross-encoder — a model that scores a query and a document together instead of independently, at the cost of being too slow to run over an entire corpus (which is why it reranks a short candidate list instead of replacing retrieval entirely). It fixes this part’s tie cleanly, and is honest about two new failure modes of its own.
Full code for this post, and the rest of the series: github.com/Thimmasani/rag-from-scratch — see retrieval/rrf.py, compare_hybrid.py (produces the tables above).