The Transformer Deep Cuts

The transformer, one concept at a time.

The core

01

Self-attention from first principles

Queries, keys, values, and the scaled dot product.

02

Multi-head attention & the block

Heads, residuals, norms, and the feed-forward.

03

Build a GPT from scratch

A small character-level model you can run.

Modern components

04

Position encoding

Absolute, RoPE, and long-context tricks.

05

The modern block

Pre-norm, RMSNorm, and SwiGLU.

06

KV cache, MQA, GQA & sliding windows

Decode without repeating work, then shrink the cache.

07

Mixture of Experts

Top-k routing, load balancing, shared vs routed.

Beyond text decoders

08

Transformer families

Decoder, encoder, encoder-decoder, prefix-LM.

09

CLIP

Learning images and text together.

10

Vision-language models

How PaliGemma-style fusion works.

Inspired by the work of Umar Jamil, Andrej Karpathy, Mohit Iyyer, Vizuara.