Foundations: from counting words to training a neural language model.
Counting, n-grams, and perplexity.
A tiny engine that writes the backward pass for you.
Embeddings, an MLP, softmax, and cross-entropy.
Init, activations, BatchNorm, LayerNorm, RMSNorm.
Tensor-level gradients, worked on paper.
SGD, momentum, Adam, AdamW, and learning-rate schedules.
From characters to byte-level BPE.
Inspired by the work of Andrej Karpathy, Mohit Iyyer.