The transformer, one concept at a time.
The core
01Queries, keys, values, and the scaled dot product.
Heads, residuals, norms, and the feed-forward.
A small character-level model you can run.
Modern components
04Absolute, RoPE, and long-context tricks.
Pre-norm, RMSNorm, and SwiGLU.
Decode without repeating work, then shrink the cache.
Top-k routing, load balancing, shared vs routed.
Beyond text decoders
08Decoder, encoder, encoder-decoder, prefix-LM.
Learning images and text together.
How PaliGemma-style fusion works.
Inspired by the work of Umar Jamil, Andrej Karpathy, Mohit Iyyer, Vizuara.