Neurons All the Way Down

Foundations: from counting words to training a neural language model.

01

What is a language model?

Counting, n-grams, and perplexity.

02

Autograd from scratch

A tiny engine that writes the backward pass for you.

03

Neural language models

Embeddings, an MLP, softmax, and cross-entropy.

04

Keeping deep nets trainable

Init, activations, BatchNorm, LayerNorm, RMSNorm.

05

Backprop by hand

Tensor-level gradients, worked on paper.

06

Optimizers

SGD, momentum, Adam, AdamW, and learning-rate schedules.

07

Tokenization

From characters to byte-level BPE.

Inspired by the work of Andrej Karpathy, Mohit Iyyer.