FINDING NEEDLES · CHAPTER 15
Two rankers, five results each, the same query. One metric says ship A. Three say ship B. Knowing why is most of evaluation.
A customer walks into a music shop and asks for violin rosin. One assistant hands over cello rosin first, then wanders off to the strings and shoulder rests, and only at the fifth try finds the right box. Another assistant starts with strings, then hands over two violin rosins and the cello rosin in a row.
The first assistant was faster to something useful. The second found everything the customer needed, and found it early. Which one is better depends on what the customer came for. A metric is a way of writing that preference down so you can measure it on a thousand searches instead of one.
Every search metric is the same recipe. Someone decides which results are right for a query. Then you look at where the right results landed in the list, and how much you care about each position.
A judgment is one row: query, item, grade. A person reads the item and decides how well it answers the query. Here the grades are 2 for exactly right, 1 for partly right, and 0 for wrong. Anything not judged counts as 0.
shelf · ranking · faithful · online
| Query “violin rosin” | Grade |
|---|---|
| light violin rosin | 2 |
| dark violin rosin | 2 |
| cello rosin | 1 |
| violin strings, shoulder rest, rosin cloth | 0 |
Three items have grade 1 or more. For metrics that only know relevant or not, those three are the relevant set. Now the two rankers:
| Rank | Ranker A | Ranker B |
|---|---|---|
| 1 | cello rosin (1) | violin strings (0) |
| 2 | violin strings (0) | light violin rosin (2) |
| 3 | shoulder rest (0) | dark violin rosin (2) |
| 4 | rosin cloth (0) | cello rosin (1) |
| 5 | light violin rosin (2) | shoulder rest (0) |
Real judgment sets are a few hundred to a few thousand queries, each with tens of judged items. Nobody judges the whole catalogue. The usual trick is pooling: judge only what any of the systems under test returned in its top results, and treat the rest as 0.
precision asks how clean the first k results are. Recall asks how much of what exists you managed to show.
| Ranker | Relevant in top 5 | P@5 | R@5 | P@3 |
|---|---|---|---|---|
| A | 2 | 2/5 = 0.400 | 2/3 = 0.667 | 0.333 |
| B | 3 | 3/5 = 0.600 | 3/3 = 1.000 | 0.667 |
For RAG, recall@k is the number to watch first. The generator only sees the top k chunks. If the needed chunk is not in them, no prompt and no model can recover it. Ranker A never shows dark violin rosin at all.
Neither metric cares about order inside the top k. Shuffle B’s five results and P@5 and R@5 stay the same.
1 if the first result is right, 1/2 if the second is, 0 if nothing is. Then average over the log.
On “violin rosin”, A’s first result is cello rosin, grade 1, so its RR is 1.000. B’s first relevant result is at rank 2, so its RR is 0.500. Two more queries from the log, “e string” and “cello bow”, put A’s first hits at ranks 2 and 1, and B’s at 1 and 2.
| Ranker | violin rosin | e string | cello bow | MRR |
|---|---|---|---|---|
| A | 1 | 1/2 | 1 | 0.833 |
| B | 1/2 | 1 | 1/2 | 0.667 |
MRR prefers A. It only looks at the first hit, and it counts a partly right first hit as a full win. That suits a “take me straight there” box, where the user clicks one result. It is the wrong metric for a RAG context window that needs every useful chunk.
Discounted cumulative gain uses the full grades. Each result adds its grade, divided by a discount that grows with rank. Then you divide by the best score any order could reach, so the result sits between 0 and 1.
rank 1 counts in full, rank 3 counts half, rank 7 counts a third. IDCG is the same sum over the ideal order of every judged item.
nDCG prefers B, by a wide margin. B puts both grade-2 items at ranks 2 and 3. A buries its one grade-2 item at rank 5, where it counts for less than half.
Interactive · One list, four metrics
At which k does B overtake A on nDCG? Is there a k where A wins on precision?
At k = 1, A wins everything: its first result is relevant and B’s is not. From k = 2 on, B leads on nDCG. A short cut-off and a long one can rank the same two systems in opposite orders, so always report k.
Many tools use a gain of 2g − 1 instead of the grade g. With grades 0 to 2 that maps to 0, 1, 3, so an exactly right result is worth three partly right ones, not two. It sharpens the reward for the top grade. The ordering logic is the same. Just check which gain a library uses before you compare numbers across papers.
Search metrics stop at the ranked list. In RAG a model then writes an answer, and the answer can go wrong on its own. Two questions cover most of it. Faithfulness: is every claim in the answer supported by the retrieved context? Answer relevance: does the answer address what was asked?
Question: “Is the light rosin good for viola?” Retrieved context: “Light violin rosin. Made from pine resin. Suits violin and viola. Sold in a cloth box.”
The toy judge here is word matching. A claim is supported if every content word in it appears in the context. Relevance is the share of the question’s content words (light, rosin, good, viola) that the answer covers. Real tools ask a model to do both checks, but the fractions are built the same way.
| Answer | Claims supported | Faithfulness | Relevance |
|---|---|---|---|
| The light rosin suits viola. It is made from pine resin. It also suits cello. | 2 of 3 (cello is not in the context) | 0.667 | 3 of 4 = 0.750 |
| Rosin is made from pine resin. | 1 of 1 | 1.000 | 1 of 4 = 0.250 |
The first answer is helpful and invents one fact. The second is perfectly faithful and does not answer the question. You need both numbers. Either one alone rewards a bad answer.
At scale, a large model grades the answers. That is fast and cheap, and it has known biases: it tends to prefer the answer shown first, prefers longer answers, and prefers text written by a model like itself. Swap the order of pairs and keep only verdicts that survive the swap. Above all, check the judge against people on a sample.
| Answer length (words) | 12 | 45 | 9 | 30 | 52 | 15 | 8 | 20 |
|---|---|---|---|---|---|---|---|---|
| Human | pass | fail | fail | pass | fail | pass | fail | pass |
| Judge | pass | pass | fail | pass | pass | pass | fail | pass |
The judge agrees with people on 6 of 8, an agreement of 0.750. It passes 6 answers where people pass 4. Both disagreements are the two longest answers, at 45 and 52 words. That is verbosity bias, found on eight rows, before it skewed a dashboard of eight thousand.
"""Score two rankers on a music shop's search log, then grade a RAG answer.
Retrieval: precision@k, recall@k, reciprocal rank, nDCG@k from graded
judgments (2 = exactly right, 1 = partly right, 0 = wrong).
Answer: faithfulness and answer relevance with a toy word-match judge.
"""
import math
# Human judgments for the query "violin rosin". Unlisted products count as 0.
JUDGED = {
"light violin rosin": 2,
"dark violin rosin": 2,
"cello rosin": 1,
"violin strings": 0,
"shoulder rest": 0,
"rosin cloth": 0,
}
RANKER_A = ["cello rosin", "violin strings", "shoulder rest", "rosin cloth", "light violin rosin"]
RANKER_B = ["violin strings", "light violin rosin", "dark violin rosin", "cello rosin", "shoulder rest"]
# Two more queries from the log, already judged relevant (1) or not (0) per rank.
OTHER_QUERIES = {
"A": {"e string": [0, 1, 0, 0, 0], "cello bow": [1, 0, 0, 0, 0]},
"B": {"e string": [1, 0, 0, 0, 0], "cello bow": [0, 1, 0, 0, 0]},
}
def grades(ranking):
return [JUDGED.get(item, 0) for item in ranking]
def total_relevant():
"""Grade 1 or more counts as relevant for precision, recall and RR."""
return sum(1 for g in JUDGED.values() if g >= 1)
def precision_at(gs, k):
return sum(1 for g in gs[:k] if g >= 1) / k
def recall_at(gs, k, n_relevant):
return sum(1 for g in gs[:k] if g >= 1) / n_relevant
def reciprocal_rank(gs):
"""1 / rank of the first relevant result. 0 if there is none."""
for rank, g in enumerate(gs, start=1):
if g >= 1:
return 1 / rank
return 0.0
def dcg_at(gs, k):
"""Gain is the grade. Rank r is discounted by log2(r + 1)."""
return sum(g / math.log2(rank + 1) for rank, g in enumerate(gs[:k], start=1))
def ndcg_at(gs, k):
"""DCG divided by the DCG of the best possible order of every judged item."""
ideal = sorted(JUDGED.values(), reverse=True)
return dcg_at(gs, k) / dcg_at(ideal, k)
def mean_reciprocal_rank(name, ranking):
rrs = [reciprocal_rank(grades(ranking))]
rrs += [reciprocal_rank(gs) for gs in OTHER_QUERIES[name].values()]
return sum(rrs) / len(rrs)
# --- Grading the answer -------------------------------------------------
STOP = {"the", "a", "is", "it", "for", "from", "in", "and", "of"}
QUESTION = "Is the light rosin good for viola?"
CONTEXT = "Light violin rosin. Made from pine resin. Suits violin and viola. Sold in a cloth box."
# Each answer is already split into claims, one sentence each.
ANSWER_1 = ["The light rosin suits viola.", "It is made from pine resin.", "It also suits cello."]
ANSWER_2 = ["Rosin is made from pine resin."]
def words(text):
cleaned = "".join(ch if ch.isalnum() else " " for ch in text.lower())
return [w for w in cleaned.split() if w not in STOP]
def supported(claim, context=CONTEXT):
"""Toy judge: every content word of the claim appears in the context."""
found = set(words(context))
return all(w in found for w in words(claim))
def faithfulness(claims):
"""Share of the answer's claims that the context supports."""
return sum(1 for c in claims if supported(c)) / len(claims)
def answer_relevance(claims, question=QUESTION):
"""Toy judge: share of the question's content words the answer covers."""
asked = words(question)
said = {w for c in claims for w in words(c)}
return sum(1 for w in asked if w in said) / len(asked)
# --- Checking the judge against people ----------------------------------
# (answer length in words, human verdict, judge verdict)
JUDGE_LOG = [
(12, "pass", "pass"), (45, "fail", "pass"), (9, "fail", "fail"), (30, "pass", "pass"),
(52, "fail", "pass"), (15, "pass", "pass"), (8, "fail", "fail"), (20, "pass", "pass"),
]
def agreement(log=JUDGE_LOG):
return sum(1 for _, human, judge in log if human == judge) / len(log)
Normalising by the ideal of the returned list. IDCG comes from every judged item, not from re-sorting the five you returned. Sort A’s own list and you get an ideal that never includes dark violin rosin, and A’s nDCG looks far better than it is.
Reporting one metric. MRR said A, nDCG said B. Pick the metric that matches how the results are used, and report recall@k next to it for RAG.
Tuning on the test set. If you adjust the ranker until the judged queries look good, the score stops predicting anything. Keep a held-out set you only score at the end.
Trusting an unchecked judge. An LLM judge’s number looks as precise as a human one. It is not, until you have measured its agreement with people on your own data.
All three relevant items (two violin rosins and the cello rosin) are in B’s top 5.
A’s first result, cello rosin, has grade 1, which counts as relevant. RR only looks at that first hit, so A gets 1 and B gets 1/2.
2/1 + 1/log2(6) = 2 + 0.387 = 2.387, divided by 3.762 is about 0.634. Same items, better order, higher score.
Yes. “Rosin is made from pine resin” is fully supported and does not answer the question. That is why relevance is measured too.