Attention
How the nine stops mix tokens.
The computation is in multi-head attention and the KV cache. The assembled blocks are the LLaMA, Mistral, DeepSeek-V3, and Qwen3-Next chapters.
The 2017 model is multi-head attention in both stacks. GPT-2 keeps it and drops the encoder. LLaMA 3 8B uses GQA: 32 query heads share 8 KV heads, 128 KiB of cache per token. Mistral multiplies that causal mask by a band of 4,096 and stores keys in a rolling buffer. DeepSeek-V3 does not use GQA. It caches 512 latent numbers plus a 64-dim RoPE key. Qwen3-Next runs Gated DeltaNet for three layers, then one softmax GQA layer with RoPE on a quarter of the head.