The Training Ground

Pre-training and fine-tuning at scale.

01

The pre-training pipeline

Data, objectives, packing, and FIM.

02

Scaling laws

Kaplan, Chinchilla, and compute-optimal training.

03

Training a GPT fast

Mixed precision, compile, FlashAttention, DDP.

04

Inside a modern dense model

Memory, FLOPs, and cluster sizing.

05

SFT & chat templates

Turn a base model into an assistant.

06

LoRA & QLoRA

Fine-tune a small adapter, not the whole net.

Inspired by the work of Aleksa Gordić, Andrej Karpathy, Mohit Iyyer, Umar Jamil.