TRUST, BUT VERIFY · CHAPTER 01
A number on a leaderboard is a dataset, a metric and a protocol. Change any of the three, and the ranking can flip. This chapter is how to read those numbers, and how they lie.
A sprint is a stopwatch. A diving routine is a panel. A jazz solo is an argument. Language models are asked to sit all three exams under one name. Treating “the score” as one number is how leaderboards get gamed.
Pick the sport first. Multiple-choice has an answer key. Code has tests. Open-ended prose needs a judge, a rubric, or a fact checker. The rest of this chapter is what can go wrong in each sport.
A benchmark is three pieces. Miss one and you cannot reproduce the number.
judges · sports · contamination · match
The items. MMLU’s four-choice questions. GPQA’s graduate science items. HumanEval’s programming prompts. GSM8K’s grade-school word problems.
Accuracy, exact match, BLEU/ROUGE, pass@k, a preference win rate, a FactScore. Same answers, different metric, different ranking.
Decoding (greedy vs sample), reasoning budget, system prompt, tools allowed, few-shot count. These are not footnotes. They move the score.
Static sets (MMLU) freeze. Dynamic ones (a live arena of pairwise human votes) drift. Broad suites hide a model that is brilliant at one task and lost at another. Automatable metrics get optimised. Human-first metrics stay expensive.
A coding prompt. You draw \(n = 5\) independent completions. \(c = 2\) pass the tests. pass@k is the chance that a set of \(k\) of those samples contains at least one success. Unbiased estimator:
the fraction of k-subsets that are all failures, subtracted from one. If there are fewer than k failures, every subset hits a success and pass@k is 1.
| \(k\) | \(\binom{3}{k} / \binom{5}{k}\) | pass@k |
|---|---|---|
| 1 | 3 / 5 | 0.40 |
| 2 | 3 / 10 | 0.70 |
| 3 | 1 / 10 | 0.90 |
| 4 | 0 | 1.00 |
Reporting only pass@1 under-sells a model that is often right on the third try. Reporting pass@100 with a hidden pass@1 of 0.02 over-sells it. Publish both, and the \(n\) you sampled.
Static benchmarks leak. A model that saw the item, or a paraphrase of it, is taking a memory test, not a skill test. Canary strings (unique markers planted in a dataset) and isomorphic held-out sets (same skill, new numbers) are two ways to notice. GPQA was written to be hard to look up; that is a protocol choice, not a guarantee.
Goodhart’s law is the shadow: once a number becomes a target, models are trained at it. A saturated benchmark stops ranking anyone. Then you need a harder set, a live arena, or a metric the trainer cannot see.
For open-ended answers there is no exact match. A stronger model can grade a weaker one against a rubric. That scales. It also inherits taste:
Mitigations: swap the order, average several judges, score a rubric instead of a pairwise winner, and keep a human spot-check. A judge is a metric. Treat it like one, including contamination of the judge’s own training data.
A biography of a made-up potter, “Lina Voss”, comes back as four claims:
FactScore is supported atoms over all atoms: 3 / 4 = 0.75. The paragraph can read fluently at 0.75. Fluency is not the metric. The next chapter, on retrieval, is one way to give the model a source so this fraction can go up. A later chapter, on memorization, is why some of those atoms were never in the weights to begin with.
"""Unbiased pass@k estimator on a tiny coding toy. Five samples, two of them correct. The page walks k = 1..4 by hand. """ from math import comb def pass_at_k(n, c, k): """Probability that at least one of k draws from n samples is correct. Unbiased estimator from Chen et al. 2021 (HumanEval): pass@k = 1 - C(n - c, k) / C(n, k) When n - c < k every k-subset includes at least one success, so the ratio is defined as 0 and pass@k is 1. """ if k > n: raise ValueError("k cannot exceed n") if n - c < k: return 1.0 return 1.0 - comb(n - c, k) / comb(n, k) # Own toy: five completions of a two-line function, two pass the tests. N_SAMPLES = 5 N_CORRECT = 2 def table(n=N_SAMPLES, c=N_CORRECT): """pass@k for every k from 1 to n, as printed on the page.""" return [pass_at_k(n, c, k) for k in range(1, n + 1)]
Comparing scores with different protocols. Greedy vs temperature 0.8, 0-shot vs 5-shot, 512 vs 32k reasoning tokens: those are different exams.
Computing pass@k as “best of k on one run”. The unbiased estimator uses a pool of n samples. It is not “I sampled k times once”.
Treating an LLM judge as ground truth. It is a noisy metric with known biases. Calibrate it against humans on a slice.
Reading 90% on a leaked set as capability. Contamination turns a skill benchmark into a memorization benchmark. See chapter 05.
There are only 3 failures. You cannot draw 4 failures, so every 4-subset contains at least one success.
Data (the items), metric (how you score), protocol (decoding, prompt, tools, budget).
Length bias. Swap order and score a rubric if you want to blunt it.
Three of four atomic claims were supported by the source. One was not. Fluency hid the miss.