Decoding DeepSeek

Case study: what DeepSeek invented, starting from a cache that would not fit.

01

DeepSeek in one page

V2, V3, R1, and five innovations.

02

Multi-head latent attention

Compress the cache instead of sharing it.

03

MLA meets RoPE

Decoupled keys: a latent plus a short rotary key.

04

DeepSeekMoE

Shared and fine-grained experts, loss-free balancing.

05

Multi-token prediction

Train the model to look more than one step ahead.

06

Training in FP8

E4M3 for the forward pass, BF16 for the optimizer.

07

The complete blueprint

MLA, MoE, MTP and FP8 in one picture.

Inspired by the work of Vizuara.