THE TRAINING GROUND · CHAPTER 04
Follow one packed sequence through a dense decoder, then price it. The pictures show the path. The arithmetic uses Llama 3 8B.
The pictures in this chapter walk one training sequence through a modern dense decoder: tokenize and pack, embed, then a stack of pre-norm blocks, then logits. Treat them as a factory tour.
After the tour we stay in the office and price Llama 3 8B: 32 layers, 32 query heads, 8 KV heads, head dim 128, hidden 4096, 8.03B parameters, vocab 128,256. Those are the site-wide facts. The pictures may show a sibling dense model (hybrid windows, YaRN). The numbers we compute do not.
Architecture chapters already own RMSNorm, SwiGLU, RoPE and GQA. This chapter’s job is the bill: parameters, bytes, FLOPs, GPU-days.
Raw documents are tokenized and packed into a fixed-length row. Positions restart per document. A segmentation mask stops attention from crossing the join. That pipeline is chapter 01.
tour · 12Ld² · activations · batch

Each token ID selects one row of an embedding table of shape vocab × hidden. For Llama 3 8B that is 128,256 × 4096.

Those vectors then enter a for-loop of blocks. Each block has its own weights. Residuals are the skip roads. Attention is the only mix across positions. RMSNorm and the MLP stay position-wise.

RMSNorm scales by the root-mean-square of the vector, not by mean and variance. The formula and the LayerNorm contrast live in The modern block.

The MLP is a gated expansion (GeGLU / SwiGLU). Llama 3 8B uses width 14,336, not a naive 4×4096. Three matrices: gate, up, down.

Attention projects Q, K, V, splits heads, optionally norms Q and K, applies RoPE, then the score-and-mix. Llama 3 8B is GQA with 8 KV heads. Cache math is Deep Cuts 06.

Without positions, a causal mask still leaks order, but the model has no clean “how far apart” signal. RoPE rotates pairs of Q/K dimensions. YaRN is a long-context recipe on top of RoPE. Both belong in Position encoding.





Scores are \(QK^\top / \sqrt{d_h}\). A causal triangle hides the future. A document mask hides the other packed note. Some stacks mix local windows with a few global layers so a 160k context does not pay a full \(n^2\) everywhere.




After the last block, a final RMSNorm and the language-model head produce logits over 128,256 tokens. Training takes cross-entropy. Decode samples.

One attention layer, GQA:
queries and the output stay full width. Keys and values are 8 heads × 128 = 1024 wide.
That is 16,777,216 + 4,194,304 + 4,194,304 + 16,777,216 = 41,943,040 weights per layer.
SwiGLU at intermediate 14,336: three matrices of 4096 × 14,336 = 176,160,768 per layer.
32 layers, plus an untied embedding and lm-head (2 × 128,256 × 4096):
| Piece | Count |
|---|---|
| Attention × 32 | 1,342,177,280 |
| MLP × 32 | 5,637,144,576 |
| Embed + lm-head (untied) | 1,050,673,152 |
| Subtotal (norms left out) | 8,029,995,008 |

Llama 3 8B in bf16:
two tensors (K and V), 32 layers, 8 KV heads, 128 numbers each, 2 bytes each.

That 128 KiB is one number for the whole model, not “per layer then times 32 again.” A page of cache spans every layer. Do not multiply by layers twice.
Weights alone: \(8.03 \times 10^9 \times 2 \approx 15\) GiB. Training is heavier. A common AdamW layout is bf16 weights, bf16 gradients, and two fp32 moments:
weight, gradient, first moment, second moment. Using 8B as a round stand-in: 8e9 × 16 ≈ 119 GiB.
One 80 GiB GPU cannot hold that. You shard (ZeRO / FSDP) or you use more GPUs. Activations are extra; checkpointing trades compute for some of those bytes.
Same 6ND as scaling laws. Per token for the published 8.03B: \(6 \times 8.03 \times 10^9 \approx 4.82 \times 10^{10}\) FLOPs. For a hand example we round \(N\) to \(8 \times 10^9\) and take Llama 3’s 15T tokens:
six times eight billion times fifteen trillion. About 7.2 × 10^{23} FLOPs.

An A100’s advertised BF16 peak is 312 TFLOPS. Real training sees a fraction of that, the model FLOPs utilization (MFU). Take 40%.
MFU 50% would cut that to about 52 days. MFU 30% stretches it past 86. The peak number on a slide is not the schedule. The schedule is peak × MFU, then divided by the GPU count you can actually keep fed.

"""Memory, FLOPs and cluster size for a Llama-3-8B-shaped dense model.
Every constant matches the site-wide facts box: 32 layers, 32 query heads,
8 KV heads, head dim 128, hidden 4096, 8.03B params, vocab 128,256.
"""
L, H, H_KV, D_H, D, VOCAB = 32, 32, 8, 128, 4096, 128_256
N_PARAMS = 8.03e9
INTERMEDIATE = 14_336 # SwiGLU width that matches the published 8.03B
BF16 = 2
KIB, GIB = 1024, 1024 ** 3
# Hand-calc stand-in used in the worked example (rounder than 8.03B).
N_HAND = 8e9
D_TOKENS = 15e12 # Llama 3 pre-training corpus
A100_BF16_FLOPS = 312e12
MFU = 0.40
def attn_params():
"""One layer: Q and O are full width; K and V are GQA-narrow."""
q = D * (H * D_H)
k = D * (H_KV * D_H)
v = D * (H_KV * D_H)
o = (H * D_H) * D
return q + k + v + o
def mlp_params():
"""SwiGLU: gate, up, down. Three D × intermediate matrices."""
return 3 * D * INTERMEDIATE
def embed_params(tied=False):
table = VOCAB * D
return table if tied else 2 * table
def param_subtotal():
"""Layers + untied embed/lm_head. Norms are ~0.3M and ignored here."""
return L * (attn_params() + mlp_params()) + embed_params(tied=False)
def cache_bytes_per_token():
"""2 (K and V) × layers × KV heads × head dim × bytes."""
return 2 * L * H_KV * D_H * BF16
def train_flops(n_params, n_tokens):
return 6 * n_params * n_tokens
def gpu_hours(n_params, n_tokens, peak=A100_BF16_FLOPS, mfu=MFU):
return train_flops(n_params, n_tokens) / (peak * mfu) / 3600
def train_bytes(n_params):
"""bf16 weights + bf16 grads + two fp32 Adam moments = 16 bytes/param."""
return 16 * n_params
Counting cache blocks once per layer and then multiplying by layers again. The 128 KiB already includes all 32 layers. One token is 128 KiB, not 32 × 128 KiB, and not “3 blocks × 32 layers = 66.”
Sizing a cluster from peak TFLOPS. Multiply by MFU first. 40% of 312 is 125, not 312.
Using 4× hidden for a gated MLP. SwiGLU widths are chosen to keep parameter count near a 4× plain MLP, so the intermediate is about \(8/3 \cdot d\), here 14,336.
Forgetting the two Adam moments. The 15 GiB checkpoint is not the training footprint. Training is closer to 119 GiB before activations.
8 KV heads. 128 KiB per token, already including all 32 layers.
The gated width is 14,336, chosen to match a ~4× plain MLP’s parameter count, not 4 × 4096 = 16,384.
No. 8e9 × 16 bytes ≈ 119 GiB, before activations.
Useful rate is 125 TFLOPS (1.248 × 10^{14} FLOP/s).
No. That is the 66-versus-3 style double count. 128 KiB is the whole model.