DECODING DEEPSEEK · CHAPTER 05
The main model predicts the next token. One extra module, chained on that hidden state, predicts the token after that. It is never shown the answer.
Four tokens: the kettle boiled over. A usual LM at “the” only has to say “kettle”. That is an easy local guess. The interesting bit is further along: once you have committed to “boiled”, “over” is almost forced, and a model that never practised that second step can still look fine on next-token loss.
Multi-token prediction asks the same hidden state to help with “boiled” as well. Not by cheating — not by reading the target — but by seeing the token the main model just aimed at, and then naming the one after it.
The extra module is a short extra thought, not a second copy of the network. DeepSeek-V3 trains one MTP module. Independent extra heads (the 2024 Meta setup) are a different design.
Most tokens are easy. A few are choice points — the verb that decides the rest of the clause. If you only train on \(t_{i+1}\), a choice-point mistake is counted once. If you also train on \(t_{i+2}\), the same mistake is counted again through the extra loss. Hard decisions get more gradient.
At inference the extra module can draft the next token for self-speculative decoding. V3 reports an 85–90% second-token acceptance rate, about 1.8× tokens per second, when they turn this on. Training still uses the module even if you throw it away at decode time.
Meta’s heads all read \(h_i\) and independently guess \(t_{i+1}, t_{i+2}, \ldots\). DeepSeek’s module is a tiny causal block that sees both \(h_i\) and the embedding of \(t_{i+1}\):
Main model: hidden \(h_i\) (causal over the prefix), predicts \(t_{i+1}\).
RMSNorm both \(h_i\) and \(\operatorname{Emb}(t_{i+1})\), concatenate, project back to \(d_{\text{model}}\).
Run one causal transformer block \(\operatorname{TRM}\) over that whole sequence of merged states.
Shared output head predicts \(t_{i+2}\).
0.3 for the first 10T tokens, 0.1 for the remaining 4.8T. One extra loss, one extra module.
The embedding matrix and the output head are shared with the main model. Only the projection, the extra block, and two RMSNorms are new parameters.
main · merge · TRM
Teacher-forced input: \(t_0, t_1, t_2, t_3, t_4\) = the, kettle, boiled, over, quietly.
| i | Hidden | Main predicts | MTP sees Emb of | MTP predicts |
|---|---|---|---|---|
| 0 | the | \( t_{i+1} \) kettle | \( t_{i+1} \) kettle | \( t_{i+2} \) boiled |
| 1 | kettle | boiled | boiled | over |
| 2 | boiled | over | over | quietly |
The leak, the one this chapter exists to kill: feeding \(\operatorname{Emb}(\text{boiled})\) and asking for “boiled”. That is copying. The loss collapses on random data and teaches nothing. The embedding index and the target index must differ by one.
leak · right · kettle
condition on the next token, predict the one after that. For a length-T sequence there are T − 2 such training rows.
mtp_indices returns three slices. The check script asserts the embedding tokens are not equal to the targets, that the extra block is causal, and that the combined loss is \(L_{\text{main}} + 0.3\,L_{\text{MTP}}\).
"""DeepSeek sequential multi-token prediction. One extra module.
The main model at position i predicts token t_{i+1}.
The MTP module combines the hidden state h_i with the embedding of t_{i+1}
and predicts t_{i+2}. It never sees the token it is asked to predict.
DeepSeek-V3 trains one such module. The loss weight is 0.3 for the first
10T tokens, then 0.1.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
def mtp_indices(seq_len):
"""Positions that have a next token to condition on and a target after that.
For a sequence of length T, valid i are 0 .. T-3, which is T-2 of them.
Hidden at i, embedding of token i+1, target is token i+2.
"""
# i = 0, 1, ..., T-3. Slices below are length T-2.
hidden_idx = slice(0, seq_len - 2)
next_idx = slice(1, seq_len - 1) # t_{i+1} — known under teacher forcing
target_idx = slice(2, seq_len) # t_{i+2} — what the module predicts
return hidden_idx, next_idx, target_idx
class CausalBlock(nn.Module):
"""One tiny pre-norm transformer block. Stands in for V3's TRM_k."""
def __init__(self, d_model, n_heads):
super().__init__()
self.n_heads = n_heads
self.d_head = d_model // n_heads
self.rms_1 = nn.RMSNorm(d_model)
self.rms_2 = nn.RMSNorm(d_model)
self.w_qkv = nn.Linear(d_model, 3 * d_model, bias=False)
self.w_o = nn.Linear(d_model, d_model, bias=False)
self.w_ff1 = nn.Linear(d_model, 4 * d_model, bias=False)
self.w_ff2 = nn.Linear(4 * d_model, d_model, bias=False)
def forward(self, x):
B, T, D = x.shape
h = self.rms_1(x)
qkv = self.w_qkv(h).view(B, T, 3, self.n_heads, self.d_head)
q, k, v = qkv.unbind(dim=2)
q, k, v = q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)
scale = self.d_head ** 0.5
scores = q @ k.transpose(-2, -1) / scale
causal = torch.triu(torch.ones(T, T, device=x.device, dtype=torch.bool), 1)
scores = scores.masked_fill(causal, float("-inf"))
ctx = torch.softmax(scores, dim=-1) @ v
ctx = ctx.transpose(1, 2).contiguous().view(B, T, D)
x = x + self.w_o(ctx)
x = x + self.w_ff2(F.silu(self.w_ff1(self.rms_2(x))))
return x
class MTPModule(nn.Module):
"""One sequential MTP head, sharing the embedding and output matrices."""
def __init__(self, d_model, n_heads, embed, lm_head):
super().__init__()
self.embed = embed
self.lm_head = lm_head
self.rms_h = nn.RMSNorm(d_model)
self.rms_e = nn.RMSNorm(d_model)
# Concatenate [norm(h_i); norm(emb_{i+1})] and mix back to d_model.
self.proj = nn.Linear(2 * d_model, d_model, bias=False)
self.trm = CausalBlock(d_model, n_heads)
def forward(self, hidden, tokens):
"""hidden and tokens are the full teacher-forced sequence (B, T, ...)."""
h_idx, next_idx, target_idx = mtp_indices(tokens.shape[1])
h = self.rms_h(hidden[:, h_idx])
# Condition on t_{i+1}. The target is t_{i+2}, a different index.
next_emb = self.rms_e(self.embed(tokens[:, next_idx]))
merged = self.proj(torch.cat([h, next_emb], dim=-1))
out = self.trm(merged)
logits = self.lm_head(out)
targets = tokens[:, target_idx]
return logits, targets
def mtp_loss(main_logits, tokens, mtp_logits, mtp_targets, lam):
"""Main next-token loss plus a weighted MTP loss. λ is 0.3, then 0.1."""
main = F.cross_entropy(
main_logits[:, :-1].contiguous().view(-1, main_logits.size(-1)),
tokens[:, 1:].contiguous().view(-1),
)
extra = F.cross_entropy(
mtp_logits.contiguous().view(-1, mtp_logits.size(-1)),
mtp_targets.contiguous().view(-1),
)
return main + lam * extra, main, extra
tokens[:, 1:-1] is \( t_{i+1} \). tokens[:, 2:] is \( t_{i+2} \). Same length, different values.CausalBlock is a stand-in for V3’s \(\operatorname{TRM}_k\). It runs over the whole merged sequence, not a length-1 dummy.The future-token leak. If the embedding you condition on is the same token you score against, the task is copy-the-input. Always check \( t_{i+1} \neq t_{i+2} \) as indices, not as “they are probably different words”.
“V3 uses two or three MTP modules.” The schematic can look like a chain of many. The report trains one MTP module.
A non-causal trunk. If position \(i\) can see \(t_{i+2}\) in the main hidden state, MTP is predicting something it has already looked at.
“MTP is not speculative decoding.” It is a training objective first. V3 also uses the extra head as a self-draft. The generic protocol lives in Beyond the Forward Pass.
It sees Emb(boiled) — that is \( t_{i+1} \) — and must predict over (\( t_{i+2} \)). Not boiled.
One.
The extra-loss weight. 0.3, then 0.1.
Otherwise a merged state at position i can attend to i+1, which already contains information about later tokens.