FINDING NEEDLES · CHAPTER 10

Hybrid search

BM25 reads the part number and misses the meaning. A dense encoder gets the meaning and misses the part number. Ask both, then merge the two lists by rank.

~15 min read6 catalogue itemsTested Python

What you'll learn

  • Why lexical search wins on exact ids and dense search wins on paraphrase
  • Why you cannot add a BM25 score to a cosine
  • How reciprocal rank fusion with k = 60 merges two ranked lists
  • How min-max weighted fusion works, and why one far-away item can reorder it

Before you start

  • BM25 — the lexical score used here
  • Dense retrieval — one vector per item, cosine to the query
  • Reranking — what usually runs after the fused list

Two clerks behind one counter

A customer walks into a hardware store and says “HX-2214 quiet hinge”. One clerk reads the part number and walks to the drawer labelled HX-2214. The other clerk does not care about labels. They hear “quiet” and think of soft-close and silent hinges.

Each clerk brings back a short list, best first. Neither list is right on its own. The first clerk also brings the HX-2214 screws. The second also brings a silent hinge of the wrong model. The item the customer wanted is second on both lists.

When two different searchers both put an item near the top, that agreement is strong evidence. Hybrid search rewards agreement. It merges positions on the lists, not the raw scores, because the two clerks do not grade on the same scale.

queryHX-2214 quiet hinge
→
BM25 + densetwo ranked lists
→
fusionby rank, k = 60
→
one listthen rerank

Each one has a blind spot

Six items in the catalogue. The query is “HX-2214 quiet hinge”. The right answer is item 1: it is the HX-2214, and soft close is what “quiet” means for a hinge.

RRF · formula · robust

MANIM RRF fuses ranked lists without calibrating raw scores.

clerks · fuse · RRF · cover

MANIM Hybrid: lexical + dense clerks, fuse with RRF or weights.
ItemCatalogue text
1HX-2214 soft close cabinet hinge
2HX-2241 steel cabinet hinge
3silent door hinge with nylon bushings
4lubricant spray for squeaky hinges
5faucet washer kit for a dripping tap
6HX-2214 hinge mounting screws

BM25 (k1 = 1.2, b = 0.75, from chapter 3) only sees shared words. No item says “quiet”, so that word adds nothing. “hinges” is not “hinge” without stemming, so item 4 scores zero too. The rare token hx-2214 does the heavy lifting. Item 6 is shorter than item 1, so it wins on length.

The dense side here is a toy: hand-made 3-D vectors on the axes hinge, quiet, plumbing. The query is (1, 1, 0). Like a real encoder, it has no idea what HX-2214 means. To it, HX-2214 and HX-2241 are the same kind of string. It ranks by meaning: silent and soft-close come first.

RankBM25 listDense list (cosine)
1item 6 · 1.621item 3 · 0.995
2item 1 · 1.491item 1 · 0.990
3item 2 · 0.487item 4 · 0.965
4item 3 · 0.414item 2 · 0.781
5—item 6 · 0.697

Each retriever returns its top 5. BM25 only has four items with a score above zero. Item 1 is second on both lists. You want a merge that puts it first.

Add up positions, not scores

The BM25 scores run from 0.414 to 1.621. The cosines run from 0.697 to 0.995. On a large corpus BM25 can reach 20 or more while cosine never passes 1. Adding the two numbers mostly adds the BM25 number. The two scales were never meant to meet.

Reciprocal rank fusion throws the scores away and keeps only each item’s position on each list.

for each list the item is on, add one over (60 plus its position). An item missing from a list gets nothing from that list.

  1. Item 1 is rank 2 on both lists: 1/62 + 1/62 = 0.03226.
  2. Item 3 is rank 4 on BM25 and rank 1 on dense: 1/64 + 1/61 = 0.03202.
  3. Item 6 is rank 1 on BM25 and rank 5 on dense: 1/61 + 1/65 = 0.03178.
  4. Item 2 is rank 3 and rank 4: 1/63 + 1/64 = 0.03150. Item 4 is only on the dense list, at rank 3: 1/63 = 0.01587.
Fused rankItemRRF score
11 · HX-2214 soft close cabinet hinge0.03226
23 · silent door hinge0.03202
36 · HX-2214 mounting screws0.03178
42 · HX-2241 steel hinge0.03150
54 · lubricant spray0.01587

Item 1 was first on neither list and is first after fusion. Two second places beat a first and a fourth. The margins are small, and that is normal for RRF: with k = 60 every rank near the top is worth about the same, so being on both lists matters more than being first on one.

k sets how steep the reward is. With k = 1, rank 1 is worth 1/2 and rank 2 only 1/3. Then item 3 wins at 0.700, and items 1 and 6 tie at 0.667. A small k trusts whoever was first. The value 60 is the default from the original RRF paper, and most search engines ship it unchanged.

Put the scores on one scale first

Sometimes you do want to trust one side more, say dense for long questions. Then you go back to scores, but rescale each list to 0..1 first. Min-max does this inside each list: the best item becomes 1, the worst becomes 0.

stretch the BM25 list and the cosine list to 0..1 each, then blend them. α = 0.5 trusts both equally. An item missing from a list counts as 0 there.

With α = 0.5 and the same top-5 lists, item 1 wins with 0.938. Items 3 and 6 tie at exactly 0.500. Each is the best of one list (1) and the worst of the other (0). Min-max has erased the fact that item 6 was a decent 0.697 cosine.

Now let the dense side return its top 6 instead of top 5. The new item is the faucet washer kit, cosine 0.229. It lands at the bottom and nobody would ever click it. But it is the new minimum, so every other cosine gets rescaled. Item 6 jumps from 0 to 0.612 on the dense side, and its fused score from 0.500 to 0.806. It now beats item 3.

Dense depthWeighted fusion order (α = 0.5)RRF order (k = 60)
top 51, 3, 6, 4, 21, 3, 6, 2, 4
top 61, 6, 3, 4, 2, 51, 3, 6, 2, 4, 5

An irrelevant item at the bottom moved the top of the weighted list. RRF did not budge: adding a sixth rank does not change the first five positions. That is why RRF is the safe default, and weighted fusion is something you tune and test on your own queries.

Interactive · Two ways to merge the same lists

Move k down toward 1 and watch item 3 take the lead. Then switch the dense depth and watch only the weighted list change.

RRF

Min-max weighted

Both retrievers and both merges

hybrid.pyRRF 1 > 3 > 6 > 2 > 4
"""Hybrid search on a six-item hardware-store catalogue.

BM25 (chapter 3) catches the part number. A toy dense encoder catches
"quiet" meaning "soft close" or "silent". Reciprocal rank fusion (k = 60)
merges the two ranked lists. Min-max weighted fusion is the alternative.

The dense vectors are hand-made, not from a trained model. The axes are
(hinge, quiet, plumbing). Like a real encoder, they cannot read the part
number: HX-2214 and HX-2241 look the same to them.
"""
import math

CATALOGUE = {
    1: "HX-2214 soft close cabinet hinge",
    2: "HX-2241 steel cabinet hinge",
    3: "silent door hinge with nylon bushings",
    4: "lubricant spray for squeaky hinges",
    5: "faucet washer kit for a dripping tap",
    6: "HX-2214 hinge mounting screws",
}

QUERY = "HX-2214 quiet hinge"

# Toy encoder output. Axes: hinge, quiet, plumbing.
DENSE = {
    "query": (1.0, 1.0, 0.0),
    1: (0.8, 0.6, 0.0),
    2: (0.9, 0.1, 0.0),
    3: (0.7, 0.7, 0.1),
    4: (0.5, 0.6, 0.2),
    5: (0.1, 0.2, 0.9),
    6: (0.6, 0.0, 0.1),
}


def tokenize(text):
    """Lowercase and split on spaces. No stemming: "hinges" is not "hinge"."""
    return text.lower().split()


def bm25_scores(k1=1.2, b=0.75):
    """BM25 from chapter 3, over the whole catalogue. Zero-score items are dropped."""
    docs = {i: tokenize(t) for i, t in CATALOGUE.items()}
    n = len(docs)
    avg_len = sum(len(d) for d in docs.values()) / n
    scores = {}
    for doc_id, words in docs.items():
        total = 0.0
        for term in set(tokenize(QUERY)):
            tf = words.count(term)
            if tf == 0:
                continue
            df = sum(1 for d in docs.values() if term in d)
            idf = math.log(1 + (n - df + 0.5) / (df + 0.5))
            total += idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(words) / avg_len))
        if total > 0:
            scores[doc_id] = total
    return scores


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))


def dense_scores():
    q = DENSE["query"]
    return {doc_id: cosine(q, DENSE[doc_id]) for doc_id in CATALOGUE}


def ranked(scores, top_k=None):
    """Item ids, best first. Ties break on the smaller id."""
    order = sorted(scores, key=lambda doc_id: (-scores[doc_id], doc_id))
    return order[:top_k] if top_k else order


def rrf(rank_lists, k=60):
    """Reciprocal rank fusion: add 1 / (k + rank) from every list the item is in."""
    fused = {}
    for ranks in rank_lists:
        for position, doc_id in enumerate(ranks, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + position)
    return fused


def min_max(scores):
    """Rescale to 0..1 inside this one list. The worst item becomes 0."""
    low, high = min(scores.values()), max(scores.values())
    return {doc_id: (s - low) / (high - low) for doc_id, s in scores.items()}


def weighted_fusion(lexical, dense, alpha=0.5):
    """alpha * lexical + (1 - alpha) * dense, after min-max. Missing counts as 0."""
    lex, den = min_max(lexical), min_max(dense)
    ids = set(lex) | set(den)
    return {i: alpha * lex.get(i, 0.0) + (1 - alpha) * den.get(i, 0.0) for i in ids}


def hybrid(top_k=5, k=60):
    """The whole pipeline: two retrievers, then RRF over their top-k lists."""
    lexical = ranked(bm25_scores(), top_k)
    dense = ranked(dense_scores(), top_k)
    return ranked(rrf([lexical, dense], k))

Mistakes learners make

Adding a BM25 score to a cosine. The two numbers live on different scales. The sum mostly follows whichever one is bigger, and on a real corpus that is BM25.

Trusting min-max on a short or lopsided list. The worst item always becomes 0 and the best always becomes 1, however close they really were. One outlier at either end rescales everyone.

Expecting fusion to find what neither side retrieved. RRF only reorders items that are on at least one list. If both retrievers miss the item, fusion cannot bring it back. Widen the lists instead.

Dropping BM25 because dense “understands” the query. Part numbers, error codes, and rare names are exactly where an encoder blurs HX-2214 into HX-2241. Keep the lexical side for those.

What to remember

  • BM25 caught HX-2214 but ranked the screws first (1.621). The toy encoder caught “quiet” but ranked the wrong hinge first (0.995).
  • RRF adds 1 / (60 + rank) across lists. Item 1, second on both, wins at 0.03226 ahead of item 3 at 0.03202.
  • A smaller k trusts first places more. At k = 1, item 3 wins with 0.700.
  • Min-max weighted fusion depends on what else is in the list. One extra item at the bottom moved item 6 from 0.500 to 0.806.

Check yourself

1. Why does BM25 give item 4 nothing, though it is about hinges?

It says “hinges”, the query says “hinge”, and this tokenizer does not stem. It does not say “quiet” or “HX-2214” either.

2. Item 4 is third on the dense list. Why is it last after RRF?

It is on only one list. It gets 1/63 = 0.01587, about half of what an item on both lists collects.

3. With k = 60, what is rank 1 worth compared with rank 2?

1/61 against 1/62. They are almost equal, so being on both lists counts for more than which list put you first.

4. Why did adding the faucet kit change the weighted order but not the RRF order?

It became the new minimum cosine, so min-max rescaled every other cosine. RRF only reads positions, and the first five positions did not change.

Go to the source

Keep going