THE TRAINING GROUND · CHAPTER 03
The math of a training step is simple. The time is spent waiting on memory. Mixed precision, compile, FlashAttention and DDP are four ways to stop waiting.
torch.compile fuses, and why that helpsA modern GPU has very fast tensor cores and a much slower road to high-bandwidth memory. A naive training step writes every intermediate — attention scores, GELU activations, separate elementwise kernels — out to that road and reads them back. The cores sit idle.
The four tricks in this chapter all shrink traffic: fewer bytes per number, fewer kernel launches, never materializing a giant score matrix, and splitting the batch across GPUs without extra chatter on every micro-step.
If a change does not cut bytes moved, it will not cut step time. Peak TFLOPS on a slide are not step time. That is why a kernel that does more math (FlashAttention) can still be faster.
Anchor config: GPT-2 124M. 12 layers, 12 heads, head dim 64, hidden 768, context 1024, vocab 50,257. At random init the next-token loss should sit near \(\ln 50257 \approx 10.82\). If it does not, the head is not wired to a uniform guess.
I/O · packing · precision · tok/s
TF32 keeps values stored as float32 but lets Ampere-and-later tensor cores multiply them with a 10-bit mantissa. You turn it on with torch.set_float32_matmul_precision("high"). Nothing about your dtypes changes.
bf16 autocast runs the big matmuls in bfloat16. bf16 keeps the float32 exponent, so the range is wide and you usually skip a loss scaler. fp16 does not: it needs a scaler so tiny gradients do not underflow. Wrap the forward and the loss. Leave the optimizer state in float32.
storage and arithmetic are separate knobs. TF32 changes only the multiply. Autocast also changes what the matmul reads.
model = torch.compile(model) traces the forward graph and fuses elementwise ops. A GELU is several small kernels if you run it eagerly: multiply, tanh, multiply again. Each kernel reads and writes HBM. A fused kernel keeps the intermediates in registers or SRAM.
The first steps pay a compile cost. Steady-state steps get the win. Compile does not change the math. It changes how many times you walk to memory to do the math.
Textbook attention builds \(QK^\top / \sqrt{d_h}\), a tensor of shape \((B, H, T, T)\), then softmax, then times \(V\). For GPT-2 124M with batch 16 and \(T = 1024\), those scores alone are
sixteen sequences, twelve heads, a 1024 by 1024 grid, two bytes each. That is 384 MiB written and read every layer, every step.
FlashAttention tiles the same product and keeps a running softmax (the “online” max and sum) in SRAM. The \(T \times T\) grid never lands in HBM. The output matches the textbook formula, which the check script confirms against a manual causal softmax.
In PyTorch you do not call a special op by hand. F.scaled_dot_product_attention(..., is_causal=True) picks a FlashAttention-class kernel when the hardware allows it.
Distributed data parallel copies the model to every GPU. Each GPU runs a different micro-batch. After the backward pass, gradients are averaged with an all-reduce, then every replica takes the same AdamW step.
The logical batch is often larger than one GPU’s memory. Gradient accumulation splits it: run \(k\) micro-steps, scale each loss by \(1/k\), and step once. With DDP you should all-reduce only on the last micro-step (no_sync on the others). Otherwise you pay a network round-trip \(k\) times for the same step.
sequences on one GPU, times context, times GPUs, times accumulation steps.
Worked numbers: micro-batch 4, context 1024, 8 GPUs, 2 accumulation steps. That is \(4 \times 1024 \times 8 \times 2 = 65{,}536\) tokens per optimizer step.
"""Small pieces of a fast GPT training step.
This is not a trainer. The Evolving Transformer stop on the training loop
owns the full loop. These helpers are the four ideas this chapter teaches:
mixed precision, a compile-shaped wrapper, FlashAttention via SDPA, and
how DDP plus gradient accumulation count tokens.
"""
import torch
import torch.nn.functional as F
# GPT-2 124M facts used on the page.
N_LAYER, N_HEAD, HEAD_DIM, HIDDEN = 12, 12, 64, 768
CONTEXT, VOCAB = 1024, 50_257
def score_bytes(batch, heads, seq, bytes_per=2):
"""Bytes for a materialized [B, H, T, T] attention-score tensor."""
return batch * heads * seq * seq * bytes_per
def tokens_per_step(micro_batch, context, world_size, accum_steps):
"""Tokens that contribute to one optimizer step."""
return micro_batch * context * world_size * accum_steps
def flash_attention(q, k, v):
"""Causal attention without writing the T×T scores to HBM.
PyTorch's scaled_dot_product_attention picks a FlashAttention-class
kernel when the hardware allows it. The math is the same as the
textbook softmax(QK^T / sqrt(d)) V.
"""
return F.scaled_dot_product_attention(q, k, v, is_causal=True)
def mixed_precision_forward(linear, x):
"""Run a linear layer under autocast. bf16 needs no loss scaler."""
# device_type="cpu" keeps the helper testable without a GPU.
with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
return linear(x)
def accum_loss(raw_loss, accum_steps):
"""Scale the micro-batch loss so k steps average like one big batch."""
return raw_loss / accum_steps
Launch with torchrun. Rank 0 logs. The optimizer sees model.module after the DDP wrap. Eight-GPU and one-GPU runs will not match bit for bit: the data pointer jumps by world_size · B · T, so shards do not line up.
Scaling fp16 habits onto bf16. bf16 already has the float32 exponent. A GradScaler is for fp16 underflow.
Forgetting to divide the accumulated loss by \(k\). Then a step of 4 micro-batches is 4× too large, and AdamW’s update explodes.
All-reducing every micro-step. The average is the same if you wait until the last one. The network cost is not.
Calling this chapter a trainer. There is no data loader, no cosine schedule, no checkpoint of Adam’s moments. Those belong in the Evolving Transformer training-loop stop.
It avoids writing and reading the \(T \times T\) scores in HBM. Memory traffic, not math, was the limit.
384 MiB per layer.
Usually no. bf16 shares float32’s exponent, so gradients rarely underflow the way fp16 gradients do.
65,536.
About 10.82, which is \(\ln 50257\).