DECODING DEEPSEEK · CHAPTER 06
Most of V3’s matrix multiplies run in E4M3. The format’s largest finite value is 448. The optimizer’s memory of the past stays BF16.
Number formats already covered the ruler: fewer bits, a smaller range, a scale that maps real values onto that range. Recap in one line: you divide by a scale, round onto the format, multiply the scale back.
V3’s twist is not “invent FP8”. It is: use a short ruler (E4M3, max 448) on small tiles of a GEMM, compute the scale from the tile in front of you, and keep a high-precision notebook for the few tensors that would forget if you rounded them.
Think of converting a city’s prices to a 3-bit currency. One exchange rate for the whole city fails: rent and coffee cannot share a scale. One rate per shop window works. That is tile-wise FP8.
The GEMMs that dominate training — the linear layers in attention and in the experts — run in FP8. A short list of exceptions stays wider, because they are sensitive or they accumulate error:
| Tensor | V3 precision |
|---|---|
| Linear GEMMs (Fprop, Dgrad, Wgrad) | FP8 E4M3 |
| Master weights | FP32 |
| AdamW first and second moments | BF16 |
| Weight gradients (accumulated) | FP32 |
| MoE gating, normalisation, RoPE | higher precision |
| Activations kept for backward | mostly FP8, with a few special cases |
The moments row is the one old pages got wrong. They are BF16, not FP32. Master weights stay FP32 so a long train does not drift. Dispatch communication between experts can also go FP8; that is the low-precision storage and communication half of the recipe.
A single scale for a whole activation tensor is hostage to the largest outlier. V3 computes one scale per token per 128 channels (a 1×128 tile), and one scale per 128×128 weight block.
map this tile’s biggest magnitude to E4M3’s biggest finite value, then round onto the grid.
Worked tile: numbers \([1, 2, 224, -10]\). amax = 224, scale = \(224/448 = 0.5\). After dividing, the 224 becomes 448, which E4M3 represents exactly. Reconstruct by multiplying 0.5 back. The small values in the tile ride the same scale, so they do not underflow just because some other row of the matrix had a giant outlier.
Tensor-core accumulators are short. V3 periodically promotes partial sums to FP32 (every 128 along K) so a long reduction does not rot. Scales are online: computed from the current tile, not from a delayed running max that can lag a spike.
tile · s=0.5 · 448
FP8 comes in two common flavours. E5M2 has more range. E4M3 has more precision. V3 uses E4M3 everywhere in the GEMMs, including the backward, because the tile scale already supplies a shared exponent. You can spend the bits on mantissa.
OCP float8_e4m3fn has no infinities. The largest finite value is 448 (\(1.75 \times 2^{8}\)). That is not 240, which is the max of a different E4M3 convention. The check script builds the grid and asserts 448 is on it.
Do not quantise “to FP8” with an INT8 formula (scale by 127 and round to an integer). FP8 has an exponent. You scale so amax → 448, then round onto the E4M3 lattice.
This is not a Tensor-Core kernel. It is the arithmetic the kernel is approximating: per-tile scale, clip to 448, nearest grid point. Moments are tagged torch.bfloat16 so the fact cannot drift.
"""DeepSeek-V3's FP8 training pieces, in miniature.
Generic number formats (IEEE 754, FP16 vs BF16, INT8) live in
Beyond the Forward Pass. This file is the DeepSeek recipe:
* E4M3 (max finite value 448) for the GEMMs
* a scale per 1×128 activation tile or 128×128 weight block
* scales computed from the current tensor, not from a stale running max
* optimizer moments stay BF16; master weights stay FP32
"""
import torch
# OCP float8_e4m3fn. Exponent 15 with mantissa 7 is NaN; 448 is the largest finite.
E4M3_MAX = 448.0
# AdamW first and second moments. DeepSeek-V3 stores these in BF16, not FP32.
MOMENT_DTYPE = torch.bfloat16
MASTER_DTYPE = torch.float32
def e4m3_grid():
"""Every non-negative finite E4M3fn value, including the extra exp=15 codes."""
vals = [0.0]
for exp in range(15):
for mant in range(8):
if exp == 0:
# Subnormals: 2^{-6} * (mant / 8) = mant / 512.
if mant:
vals.append(mant / 512.0)
else:
vals.append((1.0 + mant / 8.0) * (2.0 ** (exp - 7)))
# Extra finites: exponent all-ones, mantissa 0..6. Max is 1.75 * 2^8 = 448.
for mant in range(7):
vals.append((1.0 + mant / 8.0) * (2.0 ** 8))
return torch.tensor(vals, dtype=torch.float32)
# Built once. Used to round a real number onto the E4M3 lattice.
_GRID = None
def _grid():
global _GRID
if _GRID is None:
_GRID = e4m3_grid()
return _GRID
def round_to_e4m3(x):
"""Clip to ±448, then pick the nearest E4M3fn value (sign copied back)."""
grid = _grid().to(device=x.device)
mag = x.abs().clamp(max=E4M3_MAX)
# Distances against the positive grid; restore sign after.
idx = torch.abs(mag.unsqueeze(-1) - grid).argmin(dim=-1)
return grid[idx] * x.sign().where(x != 0, torch.ones_like(x))
def tile_scale(tile, max_finite=E4M3_MAX):
"""One scale so this tile's largest magnitude maps to 448."""
amax = tile.abs().max().clamp(min=1e-12)
return amax / max_finite
def quantize_tile(tile, max_finite=E4M3_MAX):
"""Return (fp8-rounded tile, scale) reconstructed in FP32."""
scale = tile_scale(tile, max_finite)
q = round_to_e4m3(tile / scale)
return q * scale, scale
def activation_scales(x, tile=128):
"""Per-token, per-128-channel scales. x is (tokens, channels)."""
tokens, channels = x.shape
assert channels % tile == 0
grouped = x.view(tokens, channels // tile, tile)
amax = grouped.abs().amax(dim=-1).clamp(min=1e-12) # (tokens, n_tiles)
return amax / E4M3_MAX
def weight_scales(w, block=128):
"""Per 128×128 block scales. w is (out, in)."""
out, inn = w.shape
assert out % block == 0 and inn % block == 0
blocks = w.view(out // block, block, inn // block, block)
amax = blocks.abs().amax(dim=(1, 3)).clamp(min=1e-12) # (out_blocks, in_blocks)
return amax / E4M3_MAX
E4M3_MAX = 448.0 is the number to remember. The grid builder exists so the check can see 448 is actually representable.(tokens, channels/128). Weight scales are (out/128, in/128).amax is taken on the tile you have now, not on yesterday’s batch.“E4M3 max is 240.” That is another encoding. V3 uses the OCP fn variant. Max finite is 448.
“Optimizer state is FP32.” Master weights are FP32. AdamW moments are BF16.
Quantising FP8 like INT8. There is no “times 127”. Map amax to 448, then round to an E4M3 value.
One scale per tensor. An activation outlier in one token would crush every other token’s mantissa. The 1×128 tile is the fix.
448.
BF16. Not FP32.
Two. One per 1×128 tile.
A spike in this step would be measured with last step’s amax, so you would clip or underflow. V3 computes the scale from the current tile.