THE TRAINING GROUND · CHAPTER 04

Inside a modern dense model

Follow one packed sequence through a dense decoder, then price it. The pictures show the path. The arithmetic uses Llama 3 8B.

~22 min read20 diagramsLlama 3 8B sizing

What you'll learn

  • What a packed sequence looks like as it walks a dense block
  • How to count Llama 3 8B’s parameters from the facts box
  • Why training memory is about 16 bytes per parameter
  • How \(6ND\) plus MFU becomes GPU-days

Before you start

A factory tour, then the electric bill

The pictures in this chapter walk one training sequence through a modern dense decoder: tokenize and pack, embed, then a stack of pre-norm blocks, then logits. Treat them as a factory tour.

After the tour we stay in the office and price Llama 3 8B: 32 layers, 32 query heads, 8 KV heads, head dim 128, hidden 4096, 8.03B parameters, vocab 128,256. Those are the site-wide facts. The pictures may show a sibling dense model (hybrid windows, YaRN). The numbers we compute do not.

Architecture chapters already own RMSNorm, SwiGLU, RoPE and GQA. This chapter’s job is the bill: parameters, bytes, FLOPs, GPU-days.

One sequence, one stack

Raw documents are tokenized and packed into a fixed-length row. Positions restart per document. A segmentation mask stops attention from crossing the join. That pipeline is chapter 01.

tour · 12Ld² · activations · batch

MANIM Dense GPT bill: ~12 L d² params; activations vs weights.
Tokenize two documents and pack them into one row, with per-document positions and a segmentation mask.
Tokenize two documents and pack them into one row, with per-document positions and a segmentation mask.

Each token ID selects one row of an embedding table of shape vocab × hidden. For Llama 3 8B that is 128,256 × 4096.

Token IDs index rows of the embedding table.
Token IDs index rows of the embedding table.

Those vectors then enter a for-loop of blocks. Each block has its own weights. Residuals are the skip roads. Attention is the only mix across positions. RMSNorm and the MLP stay position-wise.

The forward pass through the stack of blocks.
The forward pass through the stack of blocks.

RMSNorm scales by the root-mean-square of the vector, not by mean and variance. The formula and the LayerNorm contrast live in The modern block.

RMSNorm: scale by the root-mean-square, then a learned gain.
RMSNorm: scale by the root-mean-square, then a learned gain.

The MLP is a gated expansion (GeGLU / SwiGLU). Llama 3 8B uses width 14,336, not a naive 4×4096. Three matrices: gate, up, down.

The gated MLP: gate, up, and down projections.
The gated MLP: gate, up, and down projections.

Attention projects Q, K, V, splits heads, optionally norms Q and K, applies RoPE, then the score-and-mix. Llama 3 8B is GQA with 8 KV heads. Cache math is Deep Cuts 06.

Multi-head attention: project, split heads, rotate, score, mix.
Multi-head attention: project, split heads, rotate, score, mix.

Without positions, a causal mask still leaks order, but the model has no clean “how far apart” signal. RoPE rotates pairs of Q/K dimensions. YaRN is a long-context recipe on top of RoPE. Both belong in Position encoding.

Why the model needs positions.
Why the model needs positions.
RoPE: a rotation that depends only on the distance between two tokens.
RoPE: a rotation that depends only on the distance between two tokens.
YaRN frequency table for stretching RoPE to longer contexts.
YaRN frequency table for stretching RoPE to longer contexts.
YaRN frequencies, plotted.
YaRN frequencies, plotted.
Where YaRN enters the forward pass.
Where YaRN enters the forward pass.

Scores are \(QK^\top / \sqrt{d_h}\). A causal triangle hides the future. A document mask hides the other packed note. Some stacks mix local windows with a few global layers so a 160k context does not pay a full \(n^2\) everywhere.

Computing the attention scores.
Computing the attention scores.
Masking the scores, then mixing the values.
Masking the scores, then mixing the values.
Hybrid attention: local blocks plus global layers.
Hybrid attention: local blocks plus global layers.
Hybrid attention layout across the stack.
Hybrid attention layout across the stack.

After the last block, a final RMSNorm and the language-model head produce logits over 128,256 tokens. Training takes cross-entropy. Decode samples.

From the last block to logits and the output.
From the last block to logits and the output.

Count Llama 3 8B by hand

One attention layer, GQA:

queries and the output stay full width. Keys and values are 8 heads × 128 = 1024 wide.

That is 16,777,216 + 4,194,304 + 4,194,304 + 16,777,216 = 41,943,040 weights per layer.

SwiGLU at intermediate 14,336: three matrices of 4096 × 14,336 = 176,160,768 per layer.

32 layers, plus an untied embedding and lm-head (2 × 128,256 × 4096):

PieceCount
Attention × 321,342,177,280
MLP × 325,637,144,576
Embed + lm-head (untied)1,050,673,152
Subtotal (norms left out)8,029,995,008
RMSNorms add about 0.3M. The published total is 8.03B.
Counting the learnable parameters.
Counting the learnable parameters.

Weights, cache, and the Adam tax

Llama 3 8B in bf16:

two tensors (K and V), 32 layers, 8 KV heads, 128 numbers each, 2 bytes each.

Sizing the KV cache per token.
Sizing the KV cache per token.

That 128 KiB is one number for the whole model, not “per layer then times 32 again.” A page of cache spans every layer. Do not multiply by layers twice.

Weights alone: \(8.03 \times 10^9 \times 2 \approx 15\) GiB. Training is heavier. A common AdamW layout is bf16 weights, bf16 gradients, and two fp32 moments:

weight, gradient, first moment, second moment. Using 8B as a round stand-in: 8e9 × 16 ≈ 119 GiB.

One 80 GiB GPU cannot hold that. You shard (ZeRO / FSDP) or you use more GPUs. Activations are extra; checkpointing trades compute for some of those bytes.

Six times parameters times tokens

Same 6ND as scaling laws. Per token for the published 8.03B: \(6 \times 8.03 \times 10^9 \approx 4.82 \times 10^{10}\) FLOPs. For a hand example we round \(N\) to \(8 \times 10^9\) and take Llama 3’s 15T tokens:

six times eight billion times fifteen trillion. About 7.2 × 10^{23} FLOPs.

FLOPs per token.
FLOPs per token.

From FLOPs to GPU-days

An A100’s advertised BF16 peak is 312 TFLOPS. Real training sees a fraction of that, the model FLOPs utilization (MFU). Take 40%.

  1. Useful rate = \(0.40 \times 312 \times 10^{12} = 1.248 \times 10^{14}\) FLOP/s.
  2. GPU-seconds = \(7.2 \times 10^{23} / 1.248 \times 10^{14} \approx 5.77 \times 10^9\).
  3. GPU-hours ≈ \(1.60 \times 10^6\).
  4. On 1,024 GPUs: about 65 days.

MFU 50% would cut that to about 52 days. MFU 30% stretches it past 86. The peak number on a slide is not the schedule. The schedule is peak × MFU, then divided by the GPU count you can actually keep fed.

From total FLOPs to GPU-hours.
From total FLOPs to GPU-hours.
cluster_size.pyparams, 128 KiB cache, 16 bytes/param, 65-day cluster
"""Memory, FLOPs and cluster size for a Llama-3-8B-shaped dense model.

Every constant matches the site-wide facts box: 32 layers, 32 query heads,
8 KV heads, head dim 128, hidden 4096, 8.03B params, vocab 128,256.
"""

L, H, H_KV, D_H, D, VOCAB = 32, 32, 8, 128, 4096, 128_256
N_PARAMS = 8.03e9
INTERMEDIATE = 14_336  # SwiGLU width that matches the published 8.03B
BF16 = 2
KIB, GIB = 1024, 1024 ** 3

# Hand-calc stand-in used in the worked example (rounder than 8.03B).
N_HAND = 8e9
D_TOKENS = 15e12  # Llama 3 pre-training corpus
A100_BF16_FLOPS = 312e12
MFU = 0.40


def attn_params():
    """One layer: Q and O are full width; K and V are GQA-narrow."""
    q = D * (H * D_H)
    k = D * (H_KV * D_H)
    v = D * (H_KV * D_H)
    o = (H * D_H) * D
    return q + k + v + o


def mlp_params():
    """SwiGLU: gate, up, down. Three D × intermediate matrices."""
    return 3 * D * INTERMEDIATE


def embed_params(tied=False):
    table = VOCAB * D
    return table if tied else 2 * table


def param_subtotal():
    """Layers + untied embed/lm_head. Norms are ~0.3M and ignored here."""
    return L * (attn_params() + mlp_params()) + embed_params(tied=False)


def cache_bytes_per_token():
    """2 (K and V) × layers × KV heads × head dim × bytes."""
    return 2 * L * H_KV * D_H * BF16


def train_flops(n_params, n_tokens):
    return 6 * n_params * n_tokens


def gpu_hours(n_params, n_tokens, peak=A100_BF16_FLOPS, mfu=MFU):
    return train_flops(n_params, n_tokens) / (peak * mfu) / 3600


def train_bytes(n_params):
    """bf16 weights + bf16 grads + two fp32 Adam moments = 16 bytes/param."""
    return 16 * n_params

Mistakes learners make

Counting cache blocks once per layer and then multiplying by layers again. The 128 KiB already includes all 32 layers. One token is 128 KiB, not 32 × 128 KiB, and not “3 blocks × 32 layers = 66.”

Sizing a cluster from peak TFLOPS. Multiply by MFU first. 40% of 312 is 125, not 312.

Using 4× hidden for a gated MLP. SwiGLU widths are chosen to keep parameter count near a 4× plain MLP, so the intermediate is about \(8/3 \cdot d\), here 14,336.

Forgetting the two Adam moments. The 15 GiB checkpoint is not the training footprint. Training is closer to 119 GiB before activations.

What to remember

  • A dense decoder is embed → \(L\) pre-norm blocks → logits. Packing and masks are data, not extra layers.
  • Llama 3 8B subtotals to 8,029,995,008 weights before norms. Published size: 8.03B.
  • Cache is 128 KiB per token for the whole model. Training memory is about 16 bytes per parameter (119 GiB at 8B).
  • \(6 \times 8 \times 10^9 \times 15 \times 10^{12} = 7.2 \times 10^{23}\) FLOPs. At 40% of 312 TFLOPS on 1,024 GPUs, about 65 days.

Check yourself

1. How many KV heads does Llama 3 8B use, and what is the cache per token?

8 KV heads. 128 KiB per token, already including all 32 layers.

2. Why is the MLP 176,160,768 weights, not \(3 \times 4096 \times 16384\)?

The gated width is 14,336, chosen to match a ~4× plain MLP’s parameter count, not 4 × 4096 = 16,384.

3. Can one 80 GiB GPU hold AdamW state for 8B parameters?

No. 8e9 × 16 bytes ≈ 119 GiB, before activations.

4. What does 40% MFU do to a 312 TFLOPS peak?

Useful rate is 125 TFLOPS (1.248 × 10^{14} FLOP/s).

5. You already counted 32 layers in the cache formula. Should you multiply by 32 again for “one block per layer”?

No. That is the 66-versus-3 style double count. 128 KiB is the whole model.

Go to the source

Keep going