Case study: what DeepSeek invented, starting from a cache that would not fit.
V2, V3, R1, and five innovations.
Compress the cache instead of sharing it.
Decoupled keys: a latent plus a short rotary key.
Shared and fine-grained experts, loss-free balancing.
Train the model to look more than one step ahead.
E4M3 for the forward pass, BF16 for the optimizer.
MLA, MoE, MTP and FP8 in one picture.
Inspired by the work of Vizuara.