The Evolving Transformer

Build it: one codebase, model by model. Each stop shows only what changed.

01

The original Transformer

The 2017 encoder-decoder, post-norm.

02

GPT-2

Decoder-only, weight tying, real weight names.

03

The training loop

A trainer later stops can reuse.

04

LLaMA

Pre-norm, RMSNorm, SwiGLU, RoPE, GQA, and a KV cache.

05

Mistral → Mixtral

A sliding window, then a mixture of experts.

06

DeepSeek-V3

MLA, DeepSeekMoE, multi-token prediction.

07

Qwen3-Next

Linear attention and a hybrid stack.

08

CLIP

Two encoders and a contrastive loss.

09

PaliGemma

A prefix-LM mask over image and text tokens.

Walkthroughs merged from Deep Cuts were inspired by Umar Jamil.