DECODING DEEPSEEK · CHAPTER 01
Three models, five ideas, one verified spec. Everything else in this series is a close-up of one idea.
Imagine a library that owns 671 billion books, but a reader is only allowed to pull 37 billion of them for any one question. The building is huge. The walk for each question is short.
That is DeepSeek-V3. Most of the cost of a transformer is not “how many parameters exist”. It is “how many numbers you touch per token”. DeepSeek’s bet: store a giant sparse model, activate a dense-sized slice, and spend the savings on a long context and a cheap decode.
This series is not “how transformers work”. That lives in Deep Cuts. This series is the five changes DeepSeek made, starting from the cache bill that GQA did not shrink enough.
The architecture debut. Multi-head latent attention compresses the cache. DeepSeekMoE splits the feed-forward into many small experts plus shared ones. Routing is top-6 of 160, with two shared experts.
The model this series unpacks. Same MLA family, a larger MoE (top-8 of 256, one shared expert), one extra token-prediction module, and FP8 training. 671B total, 37B active.
Reasoning, not a new block. R1 starts from V3 and trains with reinforcement learning. The architecture chapters stop here; the reasoning series picks up R1.
When a later chapter says “DeepSeek”, it means V3 unless it names V2 or R1.
Attention, the KV cache, RoPE, MoE and number formats already have homes on this site. DeepSeek’s contributions are the five rows below. Read them in order. Chapter 07 puts them back into one block.
MLA · MoE/MTP · FP8
| Ch | Innovation | The one-line version |
|---|---|---|
| 02 | Multi-head latent attention | Cache one latent per token, not a key and value per head. |
| 03 | MLA meets RoPE | Keep a tiny shared position key so RoPE does not break absorption. |
| 04 | DeepSeekMoE | Shared expert, many fine-grained routed experts, bias instead of a fat aux loss. |
| 05 | Multi-token prediction | One extra head that looks one token further ahead. |
| 06 | Training in FP8 | E4M3 GEMMs, tile scales, BF16 optimizer moments. |
Old pages in this series disagreed with each other: top-6 vs top-8, January 2025 vs December 2024, GQA vs MLA. This table is the source of truth. Later chapters quote it. They do not invent a second copy.
| Fact | DeepSeek-V3 |
|---|---|
| Released | December 2024 |
| Parameters | 671B total, 37B active per token |
| Layers | 61 (the first is dense) |
| MoE routing | 256 routed experts + 1 shared. Top-8 (not top-6) |
| Attention | MLA, not GQA. Latent 512 + 64-dim RoPE key = 576 numbers per token per layer |
| Cache (bf16) | 70,272 bytes per token for the whole model |
| MTP | One MTP module. Loss weight λ = 0.3, then 0.1 |
| FP8 | E4M3 max finite value is 448. Optimizer moments stay BF16 |
If you remember nothing else: 671B / 37B, top-8 of 256 + 1 shared, MLA with 576 cached numbers, one MTP module, FP8 E4M3 with a 448 ceiling, released December 2024.
A V3 block is still pre-norm: attention, residual, feed-forward, residual. MLA replaces multi-head attention. DeepSeekMoE replaces the dense FFN (after the first layer). MTP hangs off the hidden state during training. FP8 is how the matrix multiplies run.
RoPE is not a fifth block. It is a 64-number slice inside MLA, which is why it gets its own chapter rather than a box of its own.
“DeepSeek uses GQA, like Llama 3.” V2 and V3 use MLA. GQA shrinks the cache by sharing heads. MLA shrinks it by compressing them. The Deep Cuts cache chapter ends with this pointer.
“V3 routes top-6.” That is V2 (160 routed, two shared, top-6). V3 is 256 routed, one shared, top-8.
“V3 came out in January 2025.” The V3 report landed in December 2024. January 2025 is R1.
“R1 is a new architecture.” It is V3 plus a training recipe. This series does not re-teach GRPO.
DeepSeek-V3. R1 keeps that architecture and changes the training. Reasoning from Scratch covers R1.
One shared expert, plus 8 of the 256 routed experts. Not top-6.
No. It uses MLA. 512 latent numbers plus a 64-dim RoPE key, 576 in total per token per layer.
The bf16 cache for one token across all 61 layers: \(576 \times 61 \times 2\).