FINDING NEEDLES · CHAPTER 10
BM25 reads the part number and misses the meaning. A dense encoder gets the meaning and misses the part number. Ask both, then merge the two lists by rank.
A customer walks into a hardware store and says “HX-2214 quiet hinge”. One clerk reads the part number and walks to the drawer labelled HX-2214. The other clerk does not care about labels. They hear “quiet” and think of soft-close and silent hinges.
Each clerk brings back a short list, best first. Neither list is right on its own. The first clerk also brings the HX-2214 screws. The second also brings a silent hinge of the wrong model. The item the customer wanted is second on both lists.
When two different searchers both put an item near the top, that agreement is strong evidence. Hybrid search rewards agreement. It merges positions on the lists, not the raw scores, because the two clerks do not grade on the same scale.
Six items in the catalogue. The query is “HX-2214 quiet hinge”. The right answer is item 1: it is the HX-2214, and soft close is what “quiet” means for a hinge.
RRF · formula · robust
clerks · fuse · RRF · cover
| Item | Catalogue text |
|---|---|
| 1 | HX-2214 soft close cabinet hinge |
| 2 | HX-2241 steel cabinet hinge |
| 3 | silent door hinge with nylon bushings |
| 4 | lubricant spray for squeaky hinges |
| 5 | faucet washer kit for a dripping tap |
| 6 | HX-2214 hinge mounting screws |
BM25 (k1 = 1.2, b = 0.75, from chapter 3) only sees shared words. No item says “quiet”, so that word adds nothing. “hinges” is not “hinge” without stemming, so item 4 scores zero too. The rare token hx-2214 does the heavy lifting. Item 6 is shorter than item 1, so it wins on length.
The dense side here is a toy: hand-made 3-D vectors on the axes hinge, quiet, plumbing. The query is (1, 1, 0). Like a real encoder, it has no idea what HX-2214 means. To it, HX-2214 and HX-2241 are the same kind of string. It ranks by meaning: silent and soft-close come first.
| Rank | BM25 list | Dense list (cosine) |
|---|---|---|
| 1 | item 6 · 1.621 | item 3 · 0.995 |
| 2 | item 1 · 1.491 | item 1 · 0.990 |
| 3 | item 2 · 0.487 | item 4 · 0.965 |
| 4 | item 3 · 0.414 | item 2 · 0.781 |
| 5 | — | item 6 · 0.697 |
Each retriever returns its top 5. BM25 only has four items with a score above zero. Item 1 is second on both lists. You want a merge that puts it first.
The BM25 scores run from 0.414 to 1.621. The cosines run from 0.697 to 0.995. On a large corpus BM25 can reach 20 or more while cosine never passes 1. Adding the two numbers mostly adds the BM25 number. The two scales were never meant to meet.
Reciprocal rank fusion throws the scores away and keeps only each item’s position on each list.
for each list the item is on, add one over (60 plus its position). An item missing from a list gets nothing from that list.
| Fused rank | Item | RRF score |
|---|---|---|
| 1 | 1 · HX-2214 soft close cabinet hinge | 0.03226 |
| 2 | 3 · silent door hinge | 0.03202 |
| 3 | 6 · HX-2214 mounting screws | 0.03178 |
| 4 | 2 · HX-2241 steel hinge | 0.03150 |
| 5 | 4 · lubricant spray | 0.01587 |
Item 1 was first on neither list and is first after fusion. Two second places beat a first and a fourth. The margins are small, and that is normal for RRF: with k = 60 every rank near the top is worth about the same, so being on both lists matters more than being first on one.
k sets how steep the reward is. With k = 1, rank 1 is worth 1/2 and rank 2 only 1/3. Then item 3 wins at 0.700, and items 1 and 6 tie at 0.667. A small k trusts whoever was first. The value 60 is the default from the original RRF paper, and most search engines ship it unchanged.
Sometimes you do want to trust one side more, say dense for long questions. Then you go back to scores, but rescale each list to 0..1 first. Min-max does this inside each list: the best item becomes 1, the worst becomes 0.
stretch the BM25 list and the cosine list to 0..1 each, then blend them. α = 0.5 trusts both equally. An item missing from a list counts as 0 there.
With α = 0.5 and the same top-5 lists, item 1 wins with 0.938. Items 3 and 6 tie at exactly 0.500. Each is the best of one list (1) and the worst of the other (0). Min-max has erased the fact that item 6 was a decent 0.697 cosine.
Now let the dense side return its top 6 instead of top 5. The new item is the faucet washer kit, cosine 0.229. It lands at the bottom and nobody would ever click it. But it is the new minimum, so every other cosine gets rescaled. Item 6 jumps from 0 to 0.612 on the dense side, and its fused score from 0.500 to 0.806. It now beats item 3.
| Dense depth | Weighted fusion order (α = 0.5) | RRF order (k = 60) |
|---|---|---|
| top 5 | 1, 3, 6, 4, 2 | 1, 3, 6, 2, 4 |
| top 6 | 1, 6, 3, 4, 2, 5 | 1, 3, 6, 2, 4, 5 |
An irrelevant item at the bottom moved the top of the weighted list. RRF did not budge: adding a sixth rank does not change the first five positions. That is why RRF is the safe default, and weighted fusion is something you tune and test on your own queries.
Interactive · Two ways to merge the same lists
Move k down toward 1 and watch item 3 take the lead. Then switch the dense depth and watch only the weighted list change.
"""Hybrid search on a six-item hardware-store catalogue.
BM25 (chapter 3) catches the part number. A toy dense encoder catches
"quiet" meaning "soft close" or "silent". Reciprocal rank fusion (k = 60)
merges the two ranked lists. Min-max weighted fusion is the alternative.
The dense vectors are hand-made, not from a trained model. The axes are
(hinge, quiet, plumbing). Like a real encoder, they cannot read the part
number: HX-2214 and HX-2241 look the same to them.
"""
import math
CATALOGUE = {
1: "HX-2214 soft close cabinet hinge",
2: "HX-2241 steel cabinet hinge",
3: "silent door hinge with nylon bushings",
4: "lubricant spray for squeaky hinges",
5: "faucet washer kit for a dripping tap",
6: "HX-2214 hinge mounting screws",
}
QUERY = "HX-2214 quiet hinge"
# Toy encoder output. Axes: hinge, quiet, plumbing.
DENSE = {
"query": (1.0, 1.0, 0.0),
1: (0.8, 0.6, 0.0),
2: (0.9, 0.1, 0.0),
3: (0.7, 0.7, 0.1),
4: (0.5, 0.6, 0.2),
5: (0.1, 0.2, 0.9),
6: (0.6, 0.0, 0.1),
}
def tokenize(text):
"""Lowercase and split on spaces. No stemming: "hinges" is not "hinge"."""
return text.lower().split()
def bm25_scores(k1=1.2, b=0.75):
"""BM25 from chapter 3, over the whole catalogue. Zero-score items are dropped."""
docs = {i: tokenize(t) for i, t in CATALOGUE.items()}
n = len(docs)
avg_len = sum(len(d) for d in docs.values()) / n
scores = {}
for doc_id, words in docs.items():
total = 0.0
for term in set(tokenize(QUERY)):
tf = words.count(term)
if tf == 0:
continue
df = sum(1 for d in docs.values() if term in d)
idf = math.log(1 + (n - df + 0.5) / (df + 0.5))
total += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(words) / avg_len))
if total > 0:
scores[doc_id] = total
return scores
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
def dense_scores():
q = DENSE["query"]
return {doc_id: cosine(q, DENSE[doc_id]) for doc_id in CATALOGUE}
def ranked(scores, top_k=None):
"""Item ids, best first. Ties break on the smaller id."""
order = sorted(scores, key=lambda doc_id: (-scores[doc_id], doc_id))
return order[:top_k] if top_k else order
def rrf(rank_lists, k=60):
"""Reciprocal rank fusion: add 1 / (k + rank) from every list the item is in."""
fused = {}
for ranks in rank_lists:
for position, doc_id in enumerate(ranks, start=1):
fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + position)
return fused
def min_max(scores):
"""Rescale to 0..1 inside this one list. The worst item becomes 0."""
low, high = min(scores.values()), max(scores.values())
return {doc_id: (s - low) / (high - low) for doc_id, s in scores.items()}
def weighted_fusion(lexical, dense, alpha=0.5):
"""alpha * lexical + (1 - alpha) * dense, after min-max. Missing counts as 0."""
lex, den = min_max(lexical), min_max(dense)
ids = set(lex) | set(den)
return {i: alpha * lex.get(i, 0.0) + (1 - alpha) * den.get(i, 0.0) for i in ids}
def hybrid(top_k=5, k=60):
"""The whole pipeline: two retrievers, then RRF over their top-k lists."""
lexical = ranked(bm25_scores(), top_k)
dense = ranked(dense_scores(), top_k)
return ranked(rrf([lexical, dense], k))
Adding a BM25 score to a cosine. The two numbers live on different scales. The sum mostly follows whichever one is bigger, and on a real corpus that is BM25.
Trusting min-max on a short or lopsided list. The worst item always becomes 0 and the best always becomes 1, however close they really were. One outlier at either end rescales everyone.
Expecting fusion to find what neither side retrieved. RRF only reorders items that are on at least one list. If both retrievers miss the item, fusion cannot bring it back. Widen the lists instead.
Dropping BM25 because dense “understands” the query. Part numbers, error codes, and rare names are exactly where an encoder blurs HX-2214 into HX-2241. Keep the lexical side for those.
It says “hinges”, the query says “hinge”, and this tokenizer does not stem. It does not say “quiet” or “HX-2214” either.
It is on only one list. It gets 1/63 = 0.01587, about half of what an item on both lists collects.
1/61 against 1/62. They are almost equal, so being on both lists counts for more than which list put you first.
It became the new minimum cosine, so min-max rescaled every other cosine. RRF only reads positions, and the first five positions did not change.