Architecture Deltas

The nine architectures The Evolving Transformer builds. Each card is only the change. Nothing else is on this page.

9stops
8changes
6maps
1family tree
§ map

The family tree

click a node to jump
Stops, in series order
§ baseline

The baseline recipe

§ timeline

The nine stops

§ deep dives

Six maps of these nine stops

Map 1 of 6

Attention

How the nine stops mix tokens.

The computation is in multi-head attention and the KV cache. The assembled blocks are the LLaMA, Mistral, DeepSeek-V3, and Qwen3-Next chapters.

The 2017 model is multi-head attention in both stacks. GPT-2 keeps it and drops the encoder. LLaMA 3 8B uses GQA: 32 query heads share 8 KV heads, 128 KiB of cache per token. Mistral multiplies that causal mask by a band of 4,096 and stores keys in a rolling buffer. DeepSeek-V3 does not use GQA. It caches 512 latent numbers plus a 64-dim RoPE key. Qwen3-Next runs Gated DeltaNet for three layers, then one softmax GQA layer with RoPE on a quarter of the head.

Map 2 of 6

Experts

Three of the nine stops replace a dense feed-forward.

The layer is in mixture of experts. DeepSeek's version is in DeepSeekMoE.

Mixtral uses 8 SwiGLU experts and a top-2 softmax router, about 46.7B total and 12.9B active. DeepSeek-V3 keeps the first layer dense, then 256 routed experts, top-8, plus one shared expert, with a sigmoid gate. Qwen3-Next puts a shared-expert MoE on every layer: 512 routed, top-10, plus one shared.

Map 3 of 6

Norm

Where the norm sits, and which norm it is.

Built in GPT-2 and LLaMA.

2017 is post-norm LayerNorm, after the residual add. GPT-2 moves it before the sublayer. LLaMA swaps LayerNorm for RMSNorm. Qwen3-Next's full-attention layers also pass Q and K through RMSNorm. CLIP's two towers are still LayerNorm.

Map 4 of 6

Position

How a token learns where it sits.

RoPE is built in LLaMA. The partial version is in Qwen3-Next.

2017 adds sinusoids. GPT-2 learns a wpe table. LLaMA 3 rotates Q and K with RoPE at θ = 500,000, and it rotates before the cache write. Qwen3-Next rotates 25% of each full-attention head, θ = 10,000,000. PaliGemma runs RoPE over the image prefix and the text. It does not add a learned position table on the concat.

Map 5 of 6

Feed-forward

The activation after attention.

SwiGLU is in LLaMA. GeGLU is in PaliGemma.

2017 uses ReLU. GPT-2 uses the tanh approximation of GELU. LLaMA, Mistral, Mixtral, and DeepSeek-V3 use SwiGLU: SiLU on the gate, times the up projection. PaliGemma's decoder uses GeGLU: GELU on the gate. Calling that SwiGLU is the mistake the chapter calls out.

Map 6 of 6

After the decoder

The two stops that are not language models, and the one hybrid.

Built in CLIP, PaliGemma, and Qwen3-Next.

CLIP matches a picture to a caption. PaliGemma writes a caption, and image tokens may see each other. Qwen3-Next mixes tokens with three Gated DeltaNet layers and then one full attention layer.

§ patterns

What these nine stops change

01

The decoder shrinks, then the cache shrinks

GPT-2 drops the encoder. LLaMA shares KV heads. Mistral caps the window. DeepSeek-V3 stores a latent instead of the heads. Qwen3-Next stops growing a cache on three layers out of four.

02

The norm moves, then changes kind

Post-norm LayerNorm, then pre-norm LayerNorm, then RMSNorm. CLIP does not take the last step.

03

Position leaves the table

Sinusoids, then a learned table, then a rotation. Qwen3-Next rotates a quarter of the head. PaliGemma rotates the image prefix and the text together.

04

SwiGLU is the language-model default here, not the only one

LLaMA, Mistral, Mixtral, and DeepSeek-V3 use SwiGLU. PaliGemma's decoder uses GeGLU. GPT-2 is still GELU.

05

Experts show up three times, differently

Mixtral is 8 large experts, top-2, softmax. DeepSeek-V3 is 256 plus a shared expert, top-8, sigmoid, and the first layer stays dense. Qwen3-Next is 512 plus a shared expert, top-10, on every layer.

06

The last two stops are not language models

CLIP scores pairs. PaliGemma generates, but the mask over the image prefix is bidirectional. Both are in the series, so both are on this page.

§ closing

What this page covers

This page is the map of one series. The original Transformer, GPT-2, LLaMA 3 8B, Mistral, Mixtral, DeepSeek-V3, Qwen3-Next, CLIP, and PaliGemma. The chapter linked on each card is where the code lives.

If a model is not one of those nine, it is not here. The atlas shows a diagram only when this folder has one: the Transformer, GPT-2, LLaMA 3 8B, DeepSeek-V3, and Qwen3-Next.