DECODING DEEPSEEK · CHAPTER 05

Multi-Token Prediction

The main model predicts the next token. One extra module, chained on that hidden state, predicts the token after that. It is never shown the answer.

~16 min readTested PyTorch codeλ = 0.3 then 0.1

What you'll learn

  • Why predicting two steps ahead densifies the training signal
  • How DeepSeek chains a module instead of using independent extra heads
  • The index rule: condition on \( t_{i+1} \), predict \( t_{i+2} \)
  • V3 trains one MTP module, with λ = 0.3 then 0.1

Before you start

The kettle is about to go

Four tokens: the kettle boiled over. A usual LM at “the” only has to say “kettle”. That is an easy local guess. The interesting bit is further along: once you have committed to “boiled”, “over” is almost forced, and a model that never practised that second step can still look fine on next-token loss.

Multi-token prediction asks the same hidden state to help with “boiled” as well. Not by cheating — not by reading the target — but by seeing the token the main model just aimed at, and then naming the one after it.

The extra module is a short extra thought, not a second copy of the network. DeepSeek-V3 trains one MTP module. Independent extra heads (the 2024 Meta setup) are a different design.

Denser signal, and a free draft head

Most tokens are easy. A few are choice points — the verb that decides the rest of the clause. If you only train on \(t_{i+1}\), a choice-point mistake is counted once. If you also train on \(t_{i+2}\), the same mistake is counted again through the extra loss. Hard decisions get more gradient.

At inference the extra module can draft the next token for self-speculative decoding. V3 reports an 85–90% second-token acceptance rate, about 1.8× tokens per second, when they turn this on. Training still uses the module even if you throw it away at decode time.

Condition on the token you just predicted

Meta’s heads all read \(h_i\) and independently guess \(t_{i+1}, t_{i+2}, \ldots\). DeepSeek’s module is a tiny causal block that sees both \(h_i\) and the embedding of \(t_{i+1}\):

  1. Main model: hidden \(h_i\) (causal over the prefix), predicts \(t_{i+1}\).

  2. RMSNorm both \(h_i\) and \(\operatorname{Emb}(t_{i+1})\), concatenate, project back to \(d_{\text{model}}\).

  3. Run one causal transformer block \(\operatorname{TRM}\) over that whole sequence of merged states.

  4. Shared output head predicts \(t_{i+2}\).

0.3 for the first 10T tokens, 0.1 for the remaining 4.8T. One extra loss, one extra module.

The embedding matrix and the output head are shared with the main model. Only the projection, the extra block, and two RMSNorms are new parameters.

main · merge · TRM

MANIM Tiny causal block conditions on the token just predicted.

Never feed a module its own target

Teacher-forced input: \(t_0, t_1, t_2, t_3, t_4\) = the, kettle, boiled, over, quietly.

iHiddenMain predictsMTP sees Emb ofMTP predicts
0the\( t_{i+1} \) kettle\( t_{i+1} \) kettle\( t_{i+2} \) boiled
1kettleboiledboiledover
2boiledoveroverquietly

The leak, the one this chapter exists to kill: feeding \(\operatorname{Emb}(\text{boiled})\) and asking for “boiled”. That is copying. The loss collapses on random data and teaches nothing. The embedding index and the target index must differ by one.

leak · right · kettle

MANIM Condition on t_{i+1}, predict t_{i+2}.

condition on the next token, predict the one after that. For a length-T sequence there are T − 2 such training rows.

Slices that cannot collapse to copying

mtp_indices returns three slices. The check script asserts the embedding tokens are not equal to the targets, that the extra block is causal, and that the combined loss is \(L_{\text{main}} + 0.3\,L_{\text{MTP}}\).

mtp.pyemb index ≠ target index; causal TRM; λ = 0.3 path
"""DeepSeek sequential multi-token prediction. One extra module.

The main model at position i predicts token t_{i+1}.
The MTP module combines the hidden state h_i with the embedding of t_{i+1}
and predicts t_{i+2}. It never sees the token it is asked to predict.

DeepSeek-V3 trains one such module. The loss weight is 0.3 for the first
10T tokens, then 0.1.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F


def mtp_indices(seq_len):
    """Positions that have a next token to condition on and a target after that.

    For a sequence of length T, valid i are 0 .. T-3, which is T-2 of them.
    Hidden at i, embedding of token i+1, target is token i+2.
    """
    # i = 0, 1, ..., T-3. Slices below are length T-2.
    hidden_idx = slice(0, seq_len - 2)
    next_idx = slice(1, seq_len - 1)     # t_{i+1}  — known under teacher forcing
    target_idx = slice(2, seq_len)       # t_{i+2}  — what the module predicts
    return hidden_idx, next_idx, target_idx


class CausalBlock(nn.Module):
    """One tiny pre-norm transformer block. Stands in for V3's TRM_k."""

    def __init__(self, d_model, n_heads):
        super().__init__()
        self.n_heads = n_heads
        self.d_head = d_model // n_heads
        self.rms_1 = nn.RMSNorm(d_model)
        self.rms_2 = nn.RMSNorm(d_model)
        self.w_qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.w_o = nn.Linear(d_model, d_model, bias=False)
        self.w_ff1 = nn.Linear(d_model, 4 * d_model, bias=False)
        self.w_ff2 = nn.Linear(4 * d_model, d_model, bias=False)

    def forward(self, x):
        B, T, D = x.shape
        h = self.rms_1(x)
        qkv = self.w_qkv(h).view(B, T, 3, self.n_heads, self.d_head)
        q, k, v = qkv.unbind(dim=2)
        q, k, v = q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)
        scale = self.d_head ** 0.5
        scores = q @ k.transpose(-2, -1) / scale
        causal = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), 1)
        scores = scores.masked_fill(causal, float("-inf"))
        ctx = torch.softmax(scores, dim=-1) @ v
        ctx = ctx.transpose(1, 2).contiguous().view(B, T, D)
        x = x + self.w_o(ctx)
        x = x + self.w_ff2(F.silu(self.w_ff1(self.rms_2(x))))
        return x


class MTPModule(nn.Module):
    """One sequential MTP head, sharing the embedding and output matrices."""

    def __init__(self, d_model, n_heads, embed, lm_head):
        super().__init__()
        self.embed = embed
        self.lm_head = lm_head
        self.rms_h = nn.RMSNorm(d_model)
        self.rms_e = nn.RMSNorm(d_model)
        # Concatenate [norm(h_i); norm(emb_{i+1})] and mix back to d_model.
        self.proj = nn.Linear(2 * d_model, d_model, bias=False)
        self.trm = CausalBlock(d_model, n_heads)

    def forward(self, hidden, tokens):
        """hidden and tokens are the full teacher-forced sequence (B, T, ...)."""
        h_idx, next_idx, target_idx = mtp_indices(tokens.shape[1])
        h = self.rms_h(hidden[:, h_idx])
        # Condition on t_{i+1}. The target is t_{i+2}, a different index.
        next_emb = self.rms_e(self.embed(tokens[:, next_idx]))
        merged = self.proj(torch.cat([h, next_emb], dim=-1))
        out = self.trm(merged)
        logits = self.lm_head(out)
        targets = tokens[:, target_idx]
        return logits, targets


def mtp_loss(main_logits, tokens, mtp_logits, mtp_targets, lam):
    """Main next-token loss plus a weighted MTP loss. λ is 0.3, then 0.1."""
    main = F.cross_entropy(
        main_logits[:, :-1].contiguous().view(-1, main_logits.size(-1)),
        tokens[:, 1:].contiguous().view(-1),
    )
    extra = F.cross_entropy(
        mtp_logits.contiguous().view(-1, mtp_logits.size(-1)),
        mtp_targets.contiguous().view(-1),
    )
    return main + lam * extra, main, extra
  • tokens[:, 1:-1] is \( t_{i+1} \). tokens[:, 2:] is \( t_{i+2} \). Same length, different values.
  • CausalBlock is a stand-in for V3’s \(\operatorname{TRM}_k\). It runs over the whole merged sequence, not a length-1 dummy.
  • The output head is the main model’s head, passed in, so the vocab projection is shared.

Mistakes learners make

The future-token leak. If the embedding you condition on is the same token you score against, the task is copy-the-input. Always check \( t_{i+1} \neq t_{i+2} \) as indices, not as “they are probably different words”.

“V3 uses two or three MTP modules.” The schematic can look like a chain of many. The report trains one MTP module.

A non-causal trunk. If position \(i\) can see \(t_{i+2}\) in the main hidden state, MTP is predicting something it has already looked at.

“MTP is not speculative decoding.” It is a training objective first. V3 also uses the extra head as a self-draft. The generic protocol lives in Beyond the Forward Pass.

What to remember

  • Main head: \(h_i \to t_{i+1}\). MTP: \((h_i, \operatorname{Emb}(t_{i+1})) \to t_{i+2}\).
  • DeepSeek chains that module (sequential). It is not a stack of independent heads.
  • V3 trains one MTP module. λ = 0.3 for 10T tokens, then 0.1.
  • The same head can draft at decode time. That is a bonus, not the training reason.

Check yourself

1. At “kettle”, what does the MTP module see, and what must it predict?

It sees Emb(boiled) — that is \( t_{i+1} \) — and must predict over (\( t_{i+2} \)). Not boiled.

2. How many MTP modules does V3 train?

One.

3. What is λ?

The extra-loss weight. 0.3, then 0.1.

4. Why must the extra block be causal?

Otherwise a merged state at position i can attend to i+1, which already contains information about later tokens.

Go to the source

Keep going