BEYOND THE FORWARD PASS · CHAPTER 04

Speculative Decoding

A small draft model guesses several tokens. The large target checks them in one parallel forward. You keep the prefix both models agree on.

~16 min read1 interactiveTested Python

What you'll learn

  • Why decode leaves compute idle on the GPU
  • The draft / target split, and what “acceptance” means
  • How a rejection still produces a bonus token
  • The arithmetic of tokens per target forward
  • How self-speculative heads reuse the same idea

Before you start

A junior writes, a senior initials

A junior types two words ahead. The senior reads both in one glance. If the first word is right, it stays. If the second is wrong, the senior crosses it out and writes their own. The pair still advanced two words, and the senior only looked once.

The target model is the source of truth. The draft is a cheap proposal. Acceptance is not “the draft is good enough”. It is “the target would have said the same token”. The output distribution matches the target, not a blend of the two.

Decode is idle on math

A decode step reads the whole model plus the cache to write one token. On a big GPU that leaves tensor cores waiting on memory. Speculative decoding spends those idle cores on a small draft, then uses one target forward to verify a block of tokens.

If the draft is usually right, you get several target-quality tokens per expensive forward. If it is usually wrong, you paid for a draft and still only got one token. The acceptance rate is the whole game.

the fox hid ran

Shared vocabulary: the, fox, hid, ran, period. Prompt: the fox. Draft length γ = 2. Both models are greedy, so you can check the table by hand.

PrefixDraft argmaxTarget argmax
the foxhidhid
the fox hid.ran
  1. Draft proposes hid, then a period.
  2. Target runs once on the prompt plus the two draft tokens. It would have said hid, then ran.
  3. hid matches. The period does not. Accept hid. Reject the period. Emit ran as the bonus token.

Tokens produced: hid, ran. Target forwards: 1. Draft forwards: 2. Without speculation the target would have needed two forwards to write the same two tokens.

prompt · draft · accept

MANIM Draft proposes hid then “.”; target keeps hid and emits ran as bonus.

Interactive · Accept the prefix

Which draft tokens survive the target’s check?

draft proposal accepted bonus from the target

Verify in parallel, keep the target’s law

The target scores every draft position in one forward because those positions are already in the prompt-plus-draft sequence. Prefill-style parallelism, used at decode time.

On a mismatch at position \(i\), you throw away the draft from \(i\) onward and sample the target’s token at \(i\) instead. That sample is free: the target already produced its distribution there. If the whole draft matches, you sample one extra bonus token from the target’s last distribution. So a step always emits at least one token, and at most \(\gamma + 1\).

α is the number of draft tokens accepted. The +1 is the bonus (or the replacement at the first mismatch).

α+1 · α · bounds

MANIM Tokens per target forward = accepted draft count plus one bonus.

With sampling, acceptance is a coin flip from the two distributions, not a greedy string match. The algorithm is written so the accepted sequence is distributed exactly as if you had sampled the target alone.

When the draft is a head of the same model

You do not have to host a second model. Multi-token prediction trains extra heads to guess the next few tokens from the same trunk. At decode time those heads are the draft. DeepSeek-V3 reports an 85–90% second-token acceptance rate this way, about 1.8× tokens per second versus the same model without it.

The serving idea is the same: propose, verify, accept a prefix. The training idea (how those heads are attached) lives in Decoding DeepSeek.

Propose, then verify

speculate.pyfox toy: accept hid, bonus ran
"""Greedy speculative decoding on a tiny shared vocabulary.

A small draft model proposes tokens. The large target model checks them
in one parallel forward. We accept the longest matching prefix and take
the target's token at the first mismatch (the bonus token).

Own example. Prompt: "the fox". Vocab: the, fox, hid, ran, .
"""

VOCAB = ["the", "fox", "hid", "ran", "."]

# Greedy next-token tables. Rows are the text so far, values are the
# model's argmax continuation. Draft is a smaller, slightly worse model.
DRAFT = {
    "the fox": "hid",
    "the fox hid": ".",
    "the fox hid .": ".",
}
TARGET = {
    "the fox": "hid",
    "the fox hid": "ran",
    "the fox hid ran": ".",
}


def propose(prompt, draft, gamma):
    """Autoregressive draft of `gamma` tokens starting from `prompt`."""
    text = prompt
    proposed = []
    for _ in range(gamma):
        tok = draft[text]
        proposed.append(tok)
        text = text + " " + tok
    return proposed


def verify(prompt, proposed, target):
    """Walk the draft prefix. Stop at the first disagreement.

    Returns (accepted, bonus). `bonus` is the target's token at the
    first mismatch, or the target's extra token after a full accept.
    """
    accepted = []
    text = prompt
    for tok in proposed:
        want = target[text]
        if tok == want:
            accepted.append(tok)
            text = text + " " + tok
        else:
            return accepted, want
    # Whole draft matched. Sample one bonus token from the target.
    bonus = target[text]
    return accepted, bonus


def speculative_step(prompt, draft=DRAFT, target=TARGET, gamma=2):
    proposed = propose(prompt, draft, gamma)
    accepted, bonus = verify(prompt, proposed, target)
    return proposed, accepted, bonus

Mistakes learners make

“The output is a mix of draft and target.” Accepted tokens passed the target’s check. The sequence is a target sequence. The draft only proposed.

Forgetting the bonus token. A full accept still samples one more token from the target. That is why the cap is γ + 1, not γ.

Drafting more tokens than the draft can hit. Long γ with a weak draft means most of the proposal is thrown away. You still paid for the draft forwards.

Confusing this with beam search. Beam keeps several full hypotheses scored by the same model. Speculation uses a cheaper model to guess, then one model to confirm.

What to remember

  • A draft proposes γ tokens. The target verifies them in one forward.
  • Keep the matching prefix. Replace the first mismatch with the target’s token.
  • A step emits between 1 and γ + 1 tokens. The extra one is the bonus.
  • The output law is the target’s. MTP is the same serving trick with heads of the same model, and about 1.8× in DeepSeek-V3.

Check yourself

1. In the fox toy, which draft token is rejected, and what replaces it?

The period is rejected. The target’s token ran replaces it. hid was already accepted.

2. How many tokens did that target forward produce?

Two: one accepted draft token plus the bonus. γ was 2, α was 1, α + 1 = 2.

3. Does a rejected token ever appear in the output?

No. Everything from the first mismatch onward is discarded. Only the target’s token at that position is kept.

4. Why does this help a memory-bound decoder?

The expensive target does one forward of a short sequence instead of one forward per token. The extra math runs on the small draft, which is cheap.

Go to the source

Keep going