DECODING DEEPSEEK · CHAPTER 07

The Complete Blueprint

One V3 block. Four inventions stacked: a compressed cache, a sparse feed-forward, an extra future head, and FP8 multiplies.

~10 min readCapstoneNo new facts

What you'll learn

  • How MLA, DeepSeekMoE, MTP and FP8 sit in one forward pass
  • What is cached, what is active, and what exists only in training
  • The same V3 facts box, one last time
  • A reading order for the V3 report

Before you start

  • Chapters 02–06 of this series

Four knobs, one token

A token walks into a V3 layer with a hidden state. Attention must remember the past without a 976 GiB cache. The feed-forward must be huge without touching 671B weights. Training must get a denser signal than next-token. The GPUs must multiply in a format whose max is 448.

Nothing in this chapter is new. If a number here disagrees with chapter 01, chapter 01 wins — and they should not disagree.

MLA + MoE + MTP + FP8

hidden state x ↓ RMSNorm MLA · cache 576 / token 512 latent + 64 RoPE key · not GQA ↓ residual, RMSNorm DeepSeekMoE · 37B of 671B 1 shared + top-8 of 256 routed ↓ residual → next layer to layer ℓ+1 MTP (train) one module · λ 0.3 then 0.1 every GEMM FP8 E4M3 · max 448
The residual stream still looks like a transformer. MLA and MoE replace the two heavy blocks. MTP reads the hidden state during training. FP8 is how the GEMMs in both blocks run.
\(x\) in
MLA (576 cached numbers) + residual
DeepSeekMoE (1 + top-8) + residual
\(x\) out · and, in training, MTP on \(h_t\)

MLA · MoE · MTP/FP8

MANIM MLA → MoE → MTP/FP8 in one pass.

What runs when

TrainingDecode
MLAfull K, V from the latentcache \(c_{KV}\) + \(k_R\); absorb \(W_{UK}\)
DeepSeekMoErouter + bias updates; 37B activesame 37B; bias frozen
MTPone module, λ = 0.3 then 0.1off, or used as a self-draft
FP8GEMMs in E4M3; moments BF16weights can stay low-precision
Cached per token: 70,272 bytes in bf16. Active per token: 37B of 671B. First of 61 layers is dense.

DeepSeek-V3, unchanged

FactDeepSeek-V3
ReleasedDecember 2024
Parameters671B total, 37B active per token
Layers61 (the first is dense)
MoE routing256 routed + 1 shared, top-8
AttentionMLA, not GQA. 512 + 64 = 576 numbers / token / layer
Cache (bf16)70,272 bytes per token
MTPOne module, λ = 0.3 then 0.1
FP8E4M3 max 448. Optimizer moments BF16

V3 report, in this series’ order

  1. §2.1.1 MLA, including decoupled RoPE — chapters 02 and 03.

  2. §2.1.2 DeepSeekMoE and loss-free balancing — chapter 04.

  3. §2.2 Multi-token prediction — chapter 05.

  4. §3.3 FP8 mixed precision — chapter 06.

R1 is the same block with a different training recipe. It is not a sixth invention in this series.

Mistakes learners make

Re-adding RoPE as a fifth pillar. Decoupled RoPE is how MLA keeps position. It is not a separate block.

Putting MTP on the decode-only path by default. It is a training loss. Using it as a draft head is optional.

Mixing V2 and V3 routing. Top-6 and two shared experts are V2. Top-8 and one shared expert are V3.

What to remember

  • MLA compresses the cache to 576 numbers per token per layer (70,272 bytes in bf16).
  • DeepSeekMoE keeps 671B on disk and 37B in the forward, via 1 shared + top-8 of 256.
  • One MTP module, λ = 0.3 then 0.1, conditions on \(t_{i+1}\) to predict \(t_{i+2}\).
  • GEMMs in FP8 E4M3 (max 448); optimizer moments BF16. Released December 2024.

Check yourself

1. Name the four inventions in the order a token meets them.

MLA (attention), DeepSeekMoE (feed-forward), MTP (training extra on the hidden state), FP8 (how the GEMMs in the first two run).

2. What is cached per token in bf16?

70,272 bytes. 576 numbers × 61 layers × 2 bytes.

3. Does decode require MTP?

No. The module can draft, or you can drop it.

4. Which model did this series study, and which one is next if you care about reasoning?

V3. R1 is V3 plus RL, in Reasoning from Scratch.

Go to the source

Keep going