BEYOND THE FORWARD PASS · CHAPTER 01
The model only ever emits a vector of logits. Greedy, temperature, top-k, top-p and beam are five ways to turn that vector into the next token.
A waiter reads you four dishes and how much the kitchen wants to cook each one. You still have to order. You can always take the favourite. You can roll a die. You can ignore anything below a cutoff. You can keep two orders in mind and decide after the next course.
That is decoding. The model writes four scores. A strategy turns those scores into one token, then repeats.
The model does not “know” the next word. It knows a distribution. Every strategy in this chapter is a rule for reading that distribution. Change the rule, and the same model tells a different story.
Prompt: The kettle began to. Tiny vocabulary: boil, whistle, cool, dance. The model’s last layer emits four logits.
exponentiate each logit, then divide by the sum, so the four numbers add to one.
With \(z = (3, 1, 0, -2)\):
| Token | \(z\) | \(e^{z}\) | \(p\) |
|---|---|---|---|
| boil | 3 | 20.086 | 0.839 |
| whistle | 1 | 2.718 | 0.114 |
| cool | 0 | 1.000 | 0.042 |
| dance | −2 | 0.135 | 0.006 |
Those four numbers are the only input every later section uses.
z · softmax · greedy
Greedy decoding picks \(\arg\max_i z_i\). Here that is boil, every time. There is no die. Two runs with the same prompt write the same continuation.
That is useful for unit tests and for short, factual answers. It is a problem for stories: the model walks the mode of the distribution and often loops (“boil boil boil”).
Greedy is a decoder, not a better model. It throws away 0.161 of the probability mass on this step alone.
Ancestral sampling draws from \(p\) as written. Temperature rescales the logits first:
divide every logit by T, then take softmax. T below 1 makes the peak taller. T above 1 flattens the table.
| \(T\) | boil | whistle | cool | dance |
|---|---|---|---|---|
| 0.5 | 0.980 | 0.018 | 0.002 | 0.000 |
| 1.0 | 0.839 | 0.114 | 0.042 | 0.006 |
| 2.0 | 0.598 | 0.220 | 0.133 | 0.049 |
\(T \to 0\) recovers greedy. \(T \to \infty\) is a uniform draw over the vocabulary. Most chat models sample near \(T = 0.7\) to \(1.0\).
T=0.5 · T=1 · T=2
Sort the logits. Keep the k highest. Set the others to \(-\infty\) so softmax puts zero there, then sample.
On the kettle, \(k = 2\) keeps boil and whistle:
| Token | After top-2 |
|---|---|
| boil | 0.881 |
| whistle | 0.119 |
| cool | 0 |
| dance | 0 |
Fixed \(k\) is blunt. In a peaked distribution you still keep a dummy. In a flat one you cut real mass. Top-p adapts the cutoff to the shape.
Sort tokens by probability, descending. Add them until the running sum is at least \(p\). Softmax over that set only.
At \(p = 0.7\), boil alone already covers 0.839, so the nucleus is one token and sampling equals greedy.
boil · whistle · renorm
Interactive · Cool, cut, sample
What happens to the kettle table as you change temperature, k and p?
A common serving stack is: divide by \(T\), apply top-k, apply top-p, then sample. Llama 3’s reference defaults are \(T = 0.6\) and \(p = 0.9\).
Greedy commits at every step. Beam search of width \(B\) keeps the \(B\) highest-scoring prefixes and expands each of them. After the last step it returns the best whole string.
New prompt: She reached for the. First-token probabilities: mug 0.45, kettle 0.40, phone 0.15.
| After token 1 | Continuations | Joint \(p\) |
|---|---|---|
| mug (0.45) | handle 0.40 · tea 0.35 · lid 0.25 | mug handle = 0.18 |
| kettle (0.40) | handle 0.80 · lid 0.15 · tea 0.05 | kettle handle = 0.32 |
Greedy picks mug, then handle. Joint probability 0.18. Beam of width 2 also keeps kettle. After step two, kettle handle at 0.32 wins. The locally worse first token was the globally better phrase.
Chat models rarely beam-search open-ended text: it is expensive, and a high joint probability is not the same as a good story. Beam still shows up in translation and in short constrained decoding.
Softmax, temperature, top-k, top-p, greedy and the two-step beam. The check script imports this file and compares it to every number on the page.
"""Greedy, temperature, top-k, top-p, and a two-step beam. The running next-token example is the prompt "The kettle began to" with a four-word vocabulary. A second example, "She reached for the", shows how beam search can beat greedy. Every number on the page comes from these functions. """ import math # Own next-token toy. Not the old softmax-wrong table. VOCAB = ["boil", "whistle", "cool", "dance"] LOGITS = [3.0, 1.0, 0.0, -2.0] def softmax(zs): """Numerically stable softmax over a list of logits.""" peak = max(zs) exps = [math.exp(z - peak) for z in zs] total = sum(exps) return [e / total for e in exps] def apply_temperature(zs, temp): """Divide logits by T. T < 1 sharpens, T > 1 flattens.""" return [z / temp for z in zs] def top_k(zs, k): """Keep the k largest logits. Set the rest to -inf before softmax.""" order = sorted(range(len(zs)), key=lambda i: zs[i], reverse=True) keep = set(order[:k]) return [zs[i] if i in keep else float("-inf") for i in range(len(zs))] def top_p(zs, p): """Nucleus sampling. Keep the smallest prefix whose probability mass >= p.""" probs = softmax(zs) order = sorted(range(len(zs)), key=lambda i: probs[i], reverse=True) kept, mass = [], 0.0 for i in order: kept.append(i) mass += probs[i] if mass >= p: break keep = set(kept) return [zs[i] if i in keep else float("-inf") for i in range(len(zs))] def greedy_token(zs, vocab=None): """Pick the single largest logit. Same result every time.""" vocab = VOCAB if vocab is None else vocab return vocab[max(range(len(zs)), key=lambda i: zs[i])] # Beam example: first-token probs, then a continuation table. # Prompt: "She reached for the" FIRST = {"mug": 0.45, "kettle": 0.40, "phone": 0.15} AFTER = { "mug": {"handle": 0.40, "tea": 0.35, "lid": 0.25}, "kettle": {"handle": 0.80, "lid": 0.15, "tea": 0.05}, } def beam_two_steps(beam_width): """Keep the top `beam_width` two-token hypotheses by joint probability.""" # Step 1: keep the best first tokens. first_ranked = sorted(FIRST.items(), key=lambda kv: kv[1], reverse=True) survivors = first_ranked[:beam_width] # Step 2: expand each survivor and keep the best joint sequences. hyps = [] for tok, p1 in survivors: for tok2, p2 in AFTER[tok].items(): hyps.append((tok, tok2, p1 * p2)) hyps.sort(key=lambda row: row[2], reverse=True) return hyps[:beam_width] def greedy_two_steps(): """Greedy: pick the best first token, then the best continuation of that token.""" tok1 = max(FIRST, key=FIRST.get) tok2 = max(AFTER[tok1], key=AFTER[tok1].get) return tok1, tok2, FIRST[tok1] * AFTER[tok1][tok2]
Reading logits as probabilities. A logit of 3 is not “80%”. Softmax first. On this page, 3 becomes 0.839, not 0.75.
“Temperature samples the top T tokens.” Temperature is a rescaling of logits. Top-k is the thing that cuts a count.
Applying top-p to logits instead of probabilities. Nucleus ranks by \(p_i\), then renormalises. Sorting the raw \(z_i\) happens to agree here, but the cutoff is a mass cutoff, not a logit cutoff.
“Beam is unused.” It is uncommon for chat. It is still the default in a lot of translation and speech.
boil. It is the largest logit (3) and the largest probability (0.839).
Dividing by 0.5 doubles every logit. The gaps grow, softmax sharpens, and boil goes from 0.839 to 0.980.
Only boil. Its mass 0.839 already exceeds 0.7, so the nucleus is one token.
mug is the better first token (0.45 vs 0.40), but kettle handle has joint probability 0.32 against mug handle’s 0.18.