TRUST, BUT VERIFY · CHAPTER 01

Evaluating LLMs

A number on a leaderboard is a dataset, a metric and a protocol. Change any of the three, and the ranking can flip. This chapter is how to read those numbers, and how they lie.

~18 min readpass@k toyTested Python

What you'll learn

  • Why a benchmark is data + metric + protocol, not a vibes score
  • How to compute the unbiased pass@k estimator by hand
  • What contamination does to a static test, and how people try to catch it
  • Where LLM-as-judge is useful, and the biases it brings
  • How FactScore breaks a generation into checkable atoms

Before you start

  • Decoding strategies (sampling changes the score)
  • A coding unit test as “the answer is right or it is not”

Three judges, three sports

A sprint is a stopwatch. A diving routine is a panel. A jazz solo is an argument. Language models are asked to sit all three exams under one name. Treating “the score” as one number is how leaderboards get gamed.

Pick the sport first. Multiple-choice has an answer key. Code has tests. Open-ended prose needs a judge, a rubric, or a fact checker. The rest of this chapter is what can go wrong in each sport.

Data, metric, protocol

A benchmark is three pieces. Miss one and you cannot reproduce the number.

judges · sports · contamination · match

MANIM Evaluation: match the judge (benchmark / prefs / online) to the claim.

Data

The items. MMLU’s four-choice questions. GPQA’s graduate science items. HumanEval’s programming prompts. GSM8K’s grade-school word problems.

Metric

Accuracy, exact match, BLEU/ROUGE, pass@k, a preference win rate, a FactScore. Same answers, different metric, different ranking.

Protocol

Decoding (greedy vs sample), reasoning budget, system prompt, tools allowed, few-shot count. These are not footnotes. They move the score.

Static sets (MMLU) freeze. Dynamic ones (a live arena of pairwise human votes) drift. Broad suites hide a model that is brilliant at one task and lost at another. Automatable metrics get optimised. Human-first metrics stay expensive.

Five samples, two of them right

A coding prompt. You draw \(n = 5\) independent completions. \(c = 2\) pass the tests. pass@k is the chance that a set of \(k\) of those samples contains at least one success. Unbiased estimator:

the fraction of k-subsets that are all failures, subtracted from one. If there are fewer than k failures, every subset hits a success and pass@k is 1.

\(k\)\(\binom{3}{k} / \binom{5}{k}\)pass@k
13 / 50.40
23 / 100.70
31 / 100.90
401.00
Own toy. n = 5, c = 2, so n − c = 3 failures. pass@1 is just the pass rate 2/5. pass@4 is 1 because you cannot pick 4 failures out of 3.

Reporting only pass@1 under-sells a model that is often right on the third try. Reporting pass@100 with a hidden pass@1 of 0.02 over-sells it. Publish both, and the \(n\) you sampled.

When the test was in the training pile

Static benchmarks leak. A model that saw the item, or a paraphrase of it, is taking a memory test, not a skill test. Canary strings (unique markers planted in a dataset) and isomorphic held-out sets (same skill, new numbers) are two ways to notice. GPQA was written to be hard to look up; that is a protocol choice, not a guarantee.

Goodhart’s law is the shadow: once a number becomes a target, models are trained at it. A saturated benchmark stops ranking anyone. Then you need a harder set, a live arena, or a metric the trainer cannot see.

Cheap, biased, still useful

For open-ended answers there is no exact match. A stronger model can grade a weaker one against a rubric. That scales. It also inherits taste:

Mitigations: swap the order, average several judges, score a rubric instead of a pairwise winner, and keep a human spot-check. A judge is a metric. Treat it like one, including contamination of the judge’s own training data.

Break the paragraph into atoms

A biography of a made-up potter, “Lina Voss”, comes back as four claims:

  1. Lina Voss is a potter. Supported by the source notes.
  2. She works in Bergen. Supported.
  3. She opened a studio in 2019. Supported.
  4. She won a Nordic craft prize in 2024. Not in any source.

FactScore is supported atoms over all atoms: 3 / 4 = 0.75. The paragraph can read fluently at 0.75. Fluency is not the metric. The next chapter, on retrieval, is one way to give the model a source so this fraction can go up. A later chapter, on memorization, is why some of those atoms were never in the weights to begin with.

The estimator you can re-run

passk.pypass@k for n = 5, c = 2
"""Unbiased pass@k estimator on a tiny coding toy.

Five samples, two of them correct. The page walks k = 1..4 by hand.
"""
from math import comb


def pass_at_k(n, c, k):
    """Probability that at least one of k draws from n samples is correct.

    Unbiased estimator from Chen et al. 2021 (HumanEval):

        pass@k = 1 - C(n - c, k) / C(n, k)

    When n - c < k every k-subset includes at least one success, so
    the ratio is defined as 0 and pass@k is 1.
    """
    if k > n:
        raise ValueError("k cannot exceed n")
    if n - c < k:
        return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)


# Own toy: five completions of a two-line function, two pass the tests.
N_SAMPLES = 5
N_CORRECT = 2


def table(n=N_SAMPLES, c=N_CORRECT):
    """pass@k for every k from 1 to n, as printed on the page."""
    return [pass_at_k(n, c, k) for k in range(1, n + 1)]

Mistakes learners make

Comparing scores with different protocols. Greedy vs temperature 0.8, 0-shot vs 5-shot, 512 vs 32k reasoning tokens: those are different exams.

Computing pass@k as “best of k on one run”. The unbiased estimator uses a pool of n samples. It is not “I sampled k times once”.

Treating an LLM judge as ground truth. It is a noisy metric with known biases. Calibrate it against humans on a slice.

Reading 90% on a leaked set as capability. Contamination turns a skill benchmark into a memorization benchmark. See chapter 05.

What to remember

  • A benchmark is data, a metric and a protocol. Name all three.
  • pass@k = 1 − C(n − c, k) / C(n, k). On the toy, 0.40, 0.70, 0.90, then 1.00.
  • Contamination and Goodhart eat static tests. Canaries and held-out isomorphs help you notice.
  • LLM judges scale and they are biased. FactScore grades atoms against a source: here, 3 / 4 = 0.75.

Check yourself

1. With n = 5 and c = 2, why is pass@4 equal to 1?

There are only 3 failures. You cannot draw 4 failures, so every 4-subset contains at least one success.

2. Name the three parts of a benchmark.

Data (the items), metric (how you score), protocol (decoding, prompt, tools, budget).

3. A judge always prefers the longer answer. Which bias is that?

Length bias. Swap order and score a rubric if you want to blunt it.

4. A fluent biography scores FactScore 0.75. What does that mean?

Three of four atomic claims were supported by the source. One was not. Fluency hid the miss.

Go to the source

Keep going