DECODING DEEPSEEK · CHAPTER 01

DeepSeek in One Page

Three models, five ideas, one verified spec. Everything else in this series is a close-up of one idea.

~12 min readCase studyVerified V3 facts

What you'll learn

  • How V2, V3 and R1 relate, and which one this series studies
  • The five things DeepSeek added to the transformer
  • A DeepSeek-V3 facts table you can trust in every later chapter
  • Where each idea sits in one transformer block

Before you start

A dense model that pretends to be smaller

Imagine a library that owns 671 billion books, but a reader is only allowed to pull 37 billion of them for any one question. The building is huge. The walk for each question is short.

That is DeepSeek-V3. Most of the cost of a transformer is not “how many parameters exist”. It is “how many numbers you touch per token”. DeepSeek’s bet: store a giant sparse model, activate a dense-sized slice, and spend the savings on a long context and a cheap decode.

This series is not “how transformers work”. That lives in Deep Cuts. This series is the five changes DeepSeek made, starting from the cache bill that GQA did not shrink enough.

V2, V3 and R1 are not three architectures

DeepSeek-V2 · May 2024

The architecture debut. Multi-head latent attention compresses the cache. DeepSeekMoE splits the feed-forward into many small experts plus shared ones. Routing is top-6 of 160, with two shared experts.

DeepSeek-V3 · December 2024

The model this series unpacks. Same MLA family, a larger MoE (top-8 of 256, one shared expert), one extra token-prediction module, and FP8 training. 671B total, 37B active.

DeepSeek-R1 · January 2025

Reasoning, not a new block. R1 starts from V3 and trains with reinforcement learning. The architecture chapters stop here; the reasoning series picks up R1.

When a later chapter says “DeepSeek”, it means V3 unless it names V2 or R1.

Each later chapter is one of these

Attention, the KV cache, RoPE, MoE and number formats already have homes on this site. DeepSeek’s contributions are the five rows below. Read them in order. Chapter 07 puts them back into one block.

MLA · MoE/MTP · FP8

MANIM MLA, RoPE, MoE, MTP, FP8 — the map of the series.
ChInnovationThe one-line version
02Multi-head latent attentionCache one latent per token, not a key and value per head.
03MLA meets RoPEKeep a tiny shared position key so RoPE does not break absorption.
04DeepSeekMoEShared expert, many fine-grained routed experts, bias instead of a fat aux loss.
05Multi-token predictionOne extra head that looks one token further ahead.
06Training in FP8E4M3 GEMMs, tile scales, BF16 optimizer moments.

DeepSeek-V3, the numbers that win

Old pages in this series disagreed with each other: top-6 vs top-8, January 2025 vs December 2024, GQA vs MLA. This table is the source of truth. Later chapters quote it. They do not invent a second copy.

FactDeepSeek-V3
ReleasedDecember 2024
Parameters671B total, 37B active per token
Layers61 (the first is dense)
MoE routing256 routed experts + 1 shared. Top-8 (not top-6)
AttentionMLA, not GQA. Latent 512 + 64-dim RoPE key = 576 numbers per token per layer
Cache (bf16)70,272 bytes per token for the whole model
MTPOne MTP module. Loss weight λ = 0.3, then 0.1
FP8E4M3 max finite value is 448. Optimizer moments stay BF16

If you remember nothing else: 671B / 37B, top-8 of 256 + 1 shared, MLA with 576 cached numbers, one MTP module, FP8 E4M3 with a 448 ceiling, released December 2024.

One block, four of the five ideas

A V3 block is still pre-norm: attention, residual, feed-forward, residual. MLA replaces multi-head attention. DeepSeekMoE replaces the dense FFN (after the first layer). MTP hangs off the hidden state during training. FP8 is how the matrix multiplies run.

hidden state \(x\)
RMSNorm → MLA (cache 576 numbers) → residualchapters 02–03
RMSNorm → DeepSeekMoE (1 shared + top-8 of 256) → residualchapter 04
to the next layer
Training extra: MTP on \(h_t\)chapter 05
Every GEMM: FP8 E4M3chapter 06

RoPE is not a fifth block. It is a 64-number slice inside MLA, which is why it gets its own chapter rather than a box of its own.

Mistakes learners make

“DeepSeek uses GQA, like Llama 3.” V2 and V3 use MLA. GQA shrinks the cache by sharing heads. MLA shrinks it by compressing them. The Deep Cuts cache chapter ends with this pointer.

“V3 routes top-6.” That is V2 (160 routed, two shared, top-6). V3 is 256 routed, one shared, top-8.

“V3 came out in January 2025.” The V3 report landed in December 2024. January 2025 is R1.

“R1 is a new architecture.” It is V3 plus a training recipe. This series does not re-teach GRPO.

What to remember

  • V2 introduced MLA and DeepSeekMoE. V3 scaled them, added MTP and FP8, and shipped in December 2024. R1 is V3 trained to reason.
  • 671B total, 37B active. 61 layers, first one dense. Top-8 of 256 routed experts plus 1 shared.
  • MLA caches 576 numbers per token per layer (70,272 bytes for the model in bf16). It does not use GQA.
  • The next five chapters are those five rows. Chapter 07 puts the block back together.

Check yourself

1. Which model does this series study in detail, and why not R1?

DeepSeek-V3. R1 keeps that architecture and changes the training. Reasoning from Scratch covers R1.

2. How many experts does a V3 token use?

One shared expert, plus 8 of the 256 routed experts. Not top-6.

3. Does V3 use GQA?

No. It uses MLA. 512 latent numbers plus a 64-dim RoPE key, 576 in total per token per layer.

4. What is 70,272 bytes?

The bf16 cache for one token across all 61 layers: \(576 \times 61 \times 2\).

Go to the source

Keep going