Pre-training and fine-tuning at scale.
Data, objectives, packing, and FIM.
Kaplan, Chinchilla, and compute-optimal training.
Mixed precision, compile, FlashAttention, DDP.
Memory, FLOPs, and cluster sizing.
Turn a base model into an assistant.
Fine-tune a small adapter, not the whole net.
Inspired by the work of Aleksa Gordić, Andrej Karpathy, Mohit Iyyer, Umar Jamil.