THE TRAINING GROUND · CHAPTER 03

Training a GPT fast

The math of a training step is simple. The time is spent waiting on memory. Mixed precision, compile, FlashAttention and DDP are four ways to stop waiting.

~18 min readGPT-2 124M numbersTested PyTorch

What you'll learn

  • Why a training step is usually memory-bound, not math-bound
  • What TF32 and bf16 autocast change — and why bf16 needs no scaler
  • What torch.compile fuses, and why that helps
  • Why FlashAttention never writes the \(T \times T\) scores to HBM
  • How DDP plus gradient accumulation count tokens

Before you start

The stove is fine. The pantry door is slow.

A modern GPU has very fast tensor cores and a much slower road to high-bandwidth memory. A naive training step writes every intermediate — attention scores, GELU activations, separate elementwise kernels — out to that road and reads them back. The cores sit idle.

The four tricks in this chapter all shrink traffic: fewer bytes per number, fewer kernel launches, never materializing a giant score matrix, and splitting the batch across GPUs without extra chatter on every micro-step.

If a change does not cut bytes moved, it will not cut step time. Peak TFLOPS on a slide are not step time. That is why a kernel that does more math (FlashAttention) can still be faster.

Same tensors, smaller multiplies

Anchor config: GPT-2 124M. 12 layers, 12 heads, head dim 64, hidden 768, context 1024, vocab 50,257. At random init the next-token loss should sit near \(\ln 50257 \approx 10.82\). If it does not, the head is not wired to a uniform guess.

I/O · packing · precision · tok/s

MANIM Fast GPT training: feed the stove; measure tokens/sec.

TF32 keeps values stored as float32 but lets Ampere-and-later tensor cores multiply them with a 10-bit mantissa. You turn it on with torch.set_float32_matmul_precision("high"). Nothing about your dtypes changes.

bf16 autocast runs the big matmuls in bfloat16. bf16 keeps the float32 exponent, so the range is wide and you usually skip a loss scaler. fp16 does not: it needs a scaler so tiny gradients do not underflow. Wrap the forward and the loss. Leave the optimizer state in float32.

storage and arithmetic are separate knobs. TF32 changes only the multiply. Autocast also changes what the matmul reads.

One trip, not eight

model = torch.compile(model) traces the forward graph and fuses elementwise ops. A GELU is several small kernels if you run it eagerly: multiply, tanh, multiply again. Each kernel reads and writes HBM. A fused kernel keeps the intermediates in registers or SRAM.

The first steps pay a compile cost. Steady-state steps get the win. Compile does not change the math. It changes how many times you walk to memory to do the math.

Never write the score matrix

Textbook attention builds \(QK^\top / \sqrt{d_h}\), a tensor of shape \((B, H, T, T)\), then softmax, then times \(V\). For GPT-2 124M with batch 16 and \(T = 1024\), those scores alone are

sixteen sequences, twelve heads, a 1024 by 1024 grid, two bytes each. That is 384 MiB written and read every layer, every step.

FlashAttention tiles the same product and keeps a running softmax (the “online” max and sum) in SRAM. The \(T \times T\) grid never lands in HBM. The output matches the textbook formula, which the check script confirms against a manual causal softmax.

In PyTorch you do not call a special op by hand. F.scaled_dot_product_attention(..., is_causal=True) picks a FlashAttention-class kernel when the hardware allows it.

Many GPUs, one optimizer step

Distributed data parallel copies the model to every GPU. Each GPU runs a different micro-batch. After the backward pass, gradients are averaged with an all-reduce, then every replica takes the same AdamW step.

The logical batch is often larger than one GPU’s memory. Gradient accumulation splits it: run \(k\) micro-steps, scale each loss by \(1/k\), and step once. With DDP you should all-reduce only on the last micro-step (no_sync on the others). Otherwise you pay a network round-trip \(k\) times for the same step.

sequences on one GPU, times context, times GPUs, times accumulation steps.

Worked numbers: micro-batch 4, context 1024, 8 GPUs, 2 accumulation steps. That is \(4 \times 1024 \times 8 \times 2 = 65{,}536\) tokens per optimizer step.

fast_step.pyscore bytes, token count, SDPA equals manual causal attention
"""Small pieces of a fast GPT training step.

This is not a trainer. The Evolving Transformer stop on the training loop
owns the full loop. These helpers are the four ideas this chapter teaches:
mixed precision, a compile-shaped wrapper, FlashAttention via SDPA, and
how DDP plus gradient accumulation count tokens.
"""
import torch
import torch.nn.functional as F

# GPT-2 124M facts used on the page.
N_LAYER, N_HEAD, HEAD_DIM, HIDDEN = 12, 12, 64, 768
CONTEXT, VOCAB = 1024, 50_257


def score_bytes(batch, heads, seq, bytes_per=2):
    """Bytes for a materialized [B, H, T, T] attention-score tensor."""
    return batch * heads * seq * seq * bytes_per


def tokens_per_step(micro_batch, context, world_size, accum_steps):
    """Tokens that contribute to one optimizer step."""
    return micro_batch * context * world_size * accum_steps


def flash_attention(q, k, v):
    """Causal attention without writing the T×T scores to HBM.

    PyTorch's scaled_dot_product_attention picks a FlashAttention-class
    kernel when the hardware allows it. The math is the same as the
    textbook softmax(QK^T / sqrt(d)) V.
    """
    return F.scaled_dot_product_attention(q, k, v, is_causal=True)


def mixed_precision_forward(linear, x):
    """Run a linear layer under autocast. bf16 needs no loss scaler."""
    # device_type="cpu" keeps the helper testable without a GPU.
    with torch.autocast(device_type="cpu", dtype=torch.bfloat16):
        return linear(x)


def accum_loss(raw_loss, accum_steps):
    """Scale the micro-batch loss so k steps average like one big batch."""
    return raw_loss / accum_steps

Launch with torchrun. Rank 0 logs. The optimizer sees model.module after the DDP wrap. Eight-GPU and one-GPU runs will not match bit for bit: the data pointer jumps by world_size · B · T, so shards do not line up.

Mistakes learners make

Scaling fp16 habits onto bf16. bf16 already has the float32 exponent. A GradScaler is for fp16 underflow.

Forgetting to divide the accumulated loss by \(k\). Then a step of 4 micro-batches is 4× too large, and AdamW’s update explodes.

All-reducing every micro-step. The average is the same if you wait until the last one. The network cost is not.

Calling this chapter a trainer. There is no data loader, no cosine schedule, no checkpoint of Adam’s moments. Those belong in the Evolving Transformer training-loop stop.

What to remember

  • Step time tracks bytes moved. Fuse, shrink dtypes, and skip the \(T \times T\) write.
  • TF32 changes the multiply. bf16 autocast also changes storage of the big ops. No scaler for bf16.
  • FlashAttention is the same softmax, streamed. 16 × 12 × 1024 × 1024 × 2 = 384 MiB of scores you do not write.
  • Tokens per step = micro-batch × T × world × accum. All-reduce on the last micro-step only.

Check yourself

1. Why can FlashAttention do more FLOPs and still be faster?

It avoids writing and reading the \(T \times T\) scores in HBM. Memory traffic, not math, was the limit.

2. GPT-2 124M, batch 16, T = 1024, bf16. How big is the score tensor?

384 MiB per layer.

3. Do you need a GradScaler with bf16?

Usually no. bf16 shares float32’s exponent, so gradients rarely underflow the way fp16 gradients do.

4. Micro-batch 4, T = 1024, 8 GPUs, 2 accum steps. Tokens per optimizer step?

65,536.

5. What should the init loss be for a 50,257-way softmax?

About 10.82, which is \(\ln 50257\).

Go to the source

Keep going