BEYOND THE FORWARD PASS · CHAPTER 04
A small draft model guesses several tokens. The large target checks them in one parallel forward. You keep the prefix both models agree on.
A junior types two words ahead. The senior reads both in one glance. If the first word is right, it stays. If the second is wrong, the senior crosses it out and writes their own. The pair still advanced two words, and the senior only looked once.
The target model is the source of truth. The draft is a cheap proposal. Acceptance is not “the draft is good enough”. It is “the target would have said the same token”. The output distribution matches the target, not a blend of the two.
A decode step reads the whole model plus the cache to write one token. On a big GPU that leaves tensor cores waiting on memory. Speculative decoding spends those idle cores on a small draft, then uses one target forward to verify a block of tokens.
If the draft is usually right, you get several target-quality tokens per expensive forward. If it is usually wrong, you paid for a draft and still only got one token. The acceptance rate is the whole game.
Shared vocabulary: the, fox, hid, ran, period. Prompt: the fox. Draft length γ = 2. Both models are greedy, so you can check the table by hand.
| Prefix | Draft argmax | Target argmax |
|---|---|---|
| the fox | hid | hid |
| the fox hid | . | ran |
Tokens produced: hid, ran. Target forwards: 1. Draft forwards: 2. Without speculation the target would have needed two forwards to write the same two tokens.
prompt · draft · accept
Interactive · Accept the prefix
Which draft tokens survive the target’s check?
The target scores every draft position in one forward because those positions are already in the prompt-plus-draft sequence. Prefill-style parallelism, used at decode time.
On a mismatch at position \(i\), you throw away the draft from \(i\) onward and sample the target’s token at \(i\) instead. That sample is free: the target already produced its distribution there. If the whole draft matches, you sample one extra bonus token from the target’s last distribution. So a step always emits at least one token, and at most \(\gamma + 1\).
α is the number of draft tokens accepted. The +1 is the bonus (or the replacement at the first mismatch).
α+1 · α · bounds
With sampling, acceptance is a coin flip from the two distributions, not a greedy string match. The algorithm is written so the accepted sequence is distributed exactly as if you had sampled the target alone.
You do not have to host a second model. Multi-token prediction trains extra heads to guess the next few tokens from the same trunk. At decode time those heads are the draft. DeepSeek-V3 reports an 85–90% second-token acceptance rate this way, about 1.8× tokens per second versus the same model without it.
The serving idea is the same: propose, verify, accept a prefix. The training idea (how those heads are attached) lives in Decoding DeepSeek.
"""Greedy speculative decoding on a tiny shared vocabulary. A small draft model proposes tokens. The large target model checks them in one parallel forward. We accept the longest matching prefix and take the target's token at the first mismatch (the bonus token). Own example. Prompt: "the fox". Vocab: the, fox, hid, ran, . """ VOCAB = ["the", "fox", "hid", "ran", "."] # Greedy next-token tables. Rows are the text so far, values are the # model's argmax continuation. Draft is a smaller, slightly worse model. DRAFT = { "the fox": "hid", "the fox hid": ".", "the fox hid .": ".", } TARGET = { "the fox": "hid", "the fox hid": "ran", "the fox hid ran": ".", } def propose(prompt, draft, gamma): """Autoregressive draft of `gamma` tokens starting from `prompt`.""" text = prompt proposed = [] for _ in range(gamma): tok = draft[text] proposed.append(tok) text = text + " " + tok return proposed def verify(prompt, proposed, target): """Walk the draft prefix. Stop at the first disagreement. Returns (accepted, bonus). `bonus` is the target's token at the first mismatch, or the target's extra token after a full accept. """ accepted = [] text = prompt for tok in proposed: want = target[text] if tok == want: accepted.append(tok) text = text + " " + tok else: return accepted, want # Whole draft matched. Sample one bonus token from the target. bonus = target[text] return accepted, bonus def speculative_step(prompt, draft=DRAFT, target=TARGET, gamma=2): proposed = propose(prompt, draft, gamma) accepted, bonus = verify(prompt, proposed, target) return proposed, accepted, bonus
“The output is a mix of draft and target.” Accepted tokens passed the target’s check. The sequence is a target sequence. The draft only proposed.
Forgetting the bonus token. A full accept still samples one more token from the target. That is why the cap is γ + 1, not γ.
Drafting more tokens than the draft can hit. Long γ with a weak draft means most of the proposal is thrown away. You still paid for the draft forwards.
Confusing this with beam search. Beam keeps several full hypotheses scored by the same model. Speculation uses a cheaper model to guess, then one model to confirm.
The period is rejected. The target’s token ran replaces it. hid was already accepted.
Two: one accepted draft token plus the bonus. γ was 2, α was 1, α + 1 = 2.
No. Everything from the first mismatch onward is discarded. Only the target’s token at that position is kept.
The expensive target does one forward of a short sequence instead of one forward per token. The extra math runs on the small draft, which is cheap.