DECODING DEEPSEEK · CHAPTER 07
One V3 block. Four inventions stacked: a compressed cache, a sparse feed-forward, an extra future head, and FP8 multiplies.
A token walks into a V3 layer with a hidden state. Attention must remember the past without a 976 GiB cache. The feed-forward must be huge without touching 671B weights. Training must get a denser signal than next-token. The GPUs must multiply in a format whose max is 448.
Nothing in this chapter is new. If a number here disagrees with chapter 01, chapter 01 wins — and they should not disagree.
MLA · MoE · MTP/FP8
| Training | Decode | |
|---|---|---|
| MLA | full K, V from the latent | cache \(c_{KV}\) + \(k_R\); absorb \(W_{UK}\) |
| DeepSeekMoE | router + bias updates; 37B active | same 37B; bias frozen |
| MTP | one module, λ = 0.3 then 0.1 | off, or used as a self-draft |
| FP8 | GEMMs in E4M3; moments BF16 | weights can stay low-precision |
| Fact | DeepSeek-V3 |
|---|---|
| Released | December 2024 |
| Parameters | 671B total, 37B active per token |
| Layers | 61 (the first is dense) |
| MoE routing | 256 routed + 1 shared, top-8 |
| Attention | MLA, not GQA. 512 + 64 = 576 numbers / token / layer |
| Cache (bf16) | 70,272 bytes per token |
| MTP | One module, λ = 0.3 then 0.1 |
| FP8 | E4M3 max 448. Optimizer moments BF16 |
§2.1.1 MLA, including decoupled RoPE — chapters 02 and 03.
§2.1.2 DeepSeekMoE and loss-free balancing — chapter 04.
§2.2 Multi-token prediction — chapter 05.
§3.3 FP8 mixed precision — chapter 06.
R1 is the same block with a different training recipe. It is not a sixth invention in this series.
Re-adding RoPE as a fifth pillar. Decoupled RoPE is how MLA keeps position. It is not a separate block.
Putting MTP on the decode-only path by default. It is a training loss. Using it as a draft head is optional.
Mixing V2 and V3 routing. Top-6 and two shared experts are V2. Top-8 and one shared expert are V3.
MLA (attention), DeepSeekMoE (feed-forward), MTP (training extra on the hidden state), FP8 (how the GEMMs in the first two run).
70,272 bytes. 576 numbers × 61 layers × 2 bytes.
No. The module can draft, or you can drop it.
V3. R1 is V3 plus RL, in Reasoning from Scratch.