THE TRAINING GROUND · CHAPTER 02
You have a fixed pile of FLOPs. Do you spend them on a bigger model, or on more tokens? Two famous papers gave opposite-sounding answers. Both are power laws. The difference is the split.
Think of the model as a kitchen and the dataset as a pantry. A huge kitchen with an empty pantry wastes the stoves. A tiny kitchen with a warehouse of food cannot cook it all before the power is cut.
Scaling laws are the recipes that say, given this much electricity, how big should the kitchen be and how full should the pantry be. They are curve fits, not proofs. Change the data mix or the architecture and the fit moves. They still beat guessing.
The only scarce thing at training time is FLOPs. Parameters \(N\) and tokens \(D\) both spend that budget. A scaling law is a rule for splitting it. (Position encodings are a different question. NoPE — training with no explicit positions — is covered in Deep Cuts 04.)
A multiply-add against one parameter, on one token, is about two FLOPs on the forward pass (the multiply and the add). The backward pass walks that multiply twice — once toward the input, once toward the weight — so about four more. That is the usual justification of the six.
20x · both · waste
kitchen · curves · Chinchilla · balance
total training FLOPs are six times parameters times tokens. Double the model or double the data, and you double the bill.
Attention’s \(T^2\) term is real, but for ordinary context lengths the matmuls with the weights dominate, so \(6ND\) is the planning number. We will use it for every example on this page.
"""Compute-optimal scaling with round numbers.
C ≈ 6 N D. The 6 is 2 FLOPs per parameter on the forward pass plus 4 on the
backward pass (the backward pass walks every multiply twice).
"""
def train_flops(n_params, n_tokens):
"""Total training FLOPs under the 6ND rule of thumb."""
return 6 * n_params * n_tokens
def tokens_per_param(n_params, n_tokens):
return n_tokens / n_params
# Starting point of the worked example: a 1B model on 20B tokens.
N0 = 1_000_000_000
D0 = 20_000_000_000
C0 = train_flops(N0, D0) # 1.2e20
# A 10× compute budget.
C10 = 10 * C0 # 1.2e21
# Kaplan's 5× / 2× split of a 10× budget.
KAPLAN_N = 5 * N0
KAPLAN_D = 2 * D0
# Chinchilla: grow N and D by the same factor. sqrt(10) ≈ 3.16.
SQRT10 = 10 ** 0.5
CHINCHILLA_N = SQRT10 * N0
CHINCHILLA_D = SQRT10 * D0
# Published (N, D) pairs quoted on the page.
GPT3_N, GPT3_D = 175_000_000_000, 300_000_000_000
GOPHER_N, GOPHER_D = 280_000_000_000, 300_000_000_000
CHINCHILLA70_N, CHINCHILLA70_D = 70_000_000_000, 1_400_000_000_000
LLAMA3_8B_N, LLAMA3_8B_D = 8.03e9, 15e12
Kaplan et al. trained many transformers, plotted test loss against \(N\), \(D\), and \(C\), and saw power laws — straight lines on log-log paper. Their planning rule, within a broad range of shapes, was: when you get more compute, grow the model faster than the data.
A compact version of that rule: a 10× compute increase is spent as about 5× parameters and 2× tokens. Five times two is ten, so \(6ND\) grows 10×. That is a new budget, not the same budget restated.
They also found that, in that range, width versus depth mattered less than raw scale, and that a larger model needed fewer tokens to reach a given loss. Those are the claims later papers pushed on.
Hoffmann et al. ran a second sweep and fitted a parametric loss
irreducible loss, plus a term that falls as the model grows, plus a term that falls as the data grows.
Under a fixed \(C \approx 6ND\), the minimum sits near equal growth: scale \(N\) and \(D\) by the same factor. A 10× budget means about \(\sqrt{10} \approx 3.16\times\) more parameters and \(3.16\times\) more tokens. That works out to roughly 20 tokens per parameter.
The headline comparison: Gopher (280B parameters, 300B tokens) versus Chinchilla (70B parameters, 1.4T tokens) on the same compute. The smaller model with more data won. Most earlier frontier models, GPT-3 included, sat well below 20 tokens per parameter. They were undertrained for their size.
Start with \(N = 1\,\text{B}\) and \(D = 20\,\text{B}\) tokens. That is already 20 tokens per parameter.
six times one billion times twenty billion. A 10× budget is 1.2 × 10^{21} FLOPs.
| Split | N | D | C = 6ND | Tokens / param |
|---|---|---|---|---|
| Start | 1B | 20B | 1.2 × 10^{20} | 20 |
| Kaplan 5× / 2× | 5B | 40B | 1.2 × 10^{21} | 8 |
| Chinchilla √10 / √10 | 3.16B | 63.2B | 1.2 × 10^{21} | 20 |
Check the Kaplan row by hand: \(6 \times 5 \times 10^9 \times 40 \times 10^9 = 1.2 \times 10^{21}\). Check Chinchilla: \(3.16 \times 3.16 = 10\), and \(20 \times 3.16 = 63.2\), so the product \(ND\) grows 10× and the ratio stays 20.
Published points on the same scale:
| Model | N | D | Tokens / param | Read |
|---|---|---|---|---|
| GPT-3 | 175B | 300B | 1.7 | undertrained |
| Gopher | 280B | 300B | 1.1 | undertrained |
| Chinchilla | 70B | 1.4T | 20 | compute-optimal |
| Llama 3 8B | 8.03B | 15T | 1,868 | overtrained on purpose |
Compute-optimal is a statement about training FLOPs. It says nothing about how many times you will run the model later. A smaller model that saw extra tokens is cheaper to serve. Llama 3 8B trained on 15T tokens — about 1,868 tokens per parameter — is far past 20:1. That is a bet that inference will dominate the bill.
Fresh tokens still beat repeats. Once the unique crawl is exhausted, teams add synthetic data or accept extra epochs. Diminishing returns set in; the power law does not stay a straight line forever.
Test-time compute (search, sampling, verifiers) is a third axis. It is not a pre-training scaling law. That story starts in Reasoning from Scratch.
“5× parameters and 2× data keeps the same FLOPs.” It spends 10× FLOPs. That is the new budget.
“Chinchilla says 20 tokens per parameter forever.” It said that for compute-optimal training under their fit. Serving economics pushed the industry past it.
“Model shape never matters.” Kaplan’s claim was “within a broad range.” Attention type, depth, and MoE routing all move the curve.
Repeating a small corpus a thousand times and calling it \(D\). Unique tokens are the pantry. Epochs are reheating leftovers.
\(1.2 \times 10^{20}\).
5B parameters, 40B tokens, 8 tokens per parameter.
About 3.16B parameters and 63.2B tokens, still 20 tokens per parameter.
No. \(15 \times 10^{12} / 8.03 \times 10^9 \approx 1{,}868\) tokens per parameter. It is overtrained, on purpose.
Because inference runs many more times than training. A smaller, extra-trained model is cheaper to serve.