DECODING DEEPSEEK · CHAPTER 06

Training in FP8

Most of V3’s matrix multiplies run in E4M3. The format’s largest finite value is 448. The optimizer’s memory of the past stays BF16.

~16 min readTested PythonE4M3 max 448

What you'll learn

  • Which tensors V3 keeps in FP8, BF16, and FP32
  • Why scales are per 1×128 activation tile and per 128×128 weight block
  • Why E4M3 (max 448) and why scales are computed online
  • That AdamW moments stay BF16 — they are not FP32

Before you start

A short ruler, used locally

Number formats already covered the ruler: fewer bits, a smaller range, a scale that maps real values onto that range. Recap in one line: you divide by a scale, round onto the format, multiply the scale back.

V3’s twist is not “invent FP8”. It is: use a short ruler (E4M3, max 448) on small tiles of a GEMM, compute the scale from the tile in front of you, and keep a high-precision notebook for the few tensors that would forget if you rounded them.

Think of converting a city’s prices to a 3-bit currency. One exchange rate for the whole city fails: rent and coffee cannot share a scale. One rate per shop window works. That is tile-wise FP8.

Not everything is FP8

The GEMMs that dominate training — the linear layers in attention and in the experts — run in FP8. A short list of exceptions stays wider, because they are sensitive or they accumulate error:

TensorV3 precision
Linear GEMMs (Fprop, Dgrad, Wgrad)FP8 E4M3
Master weightsFP32
AdamW first and second momentsBF16
Weight gradients (accumulated)FP32
MoE gating, normalisation, RoPEhigher precision
Activations kept for backwardmostly FP8, with a few special cases

The moments row is the one old pages got wrong. They are BF16, not FP32. Master weights stay FP32 so a long train does not drift. Dispatch communication between experts can also go FP8; that is the low-precision storage and communication half of the recipe.

1×128 activations, 128×128 weights

A single scale for a whole activation tensor is hostage to the largest outlier. V3 computes one scale per token per 128 channels (a 1×128 tile), and one scale per 128×128 weight block.

map this tile’s biggest magnitude to E4M3’s biggest finite value, then round onto the grid.

Worked tile: numbers \([1, 2, 224, -10]\). amax = 224, scale = \(224/448 = 0.5\). After dividing, the 224 becomes 448, which E4M3 represents exactly. Reconstruct by multiplying 0.5 back. The small values in the tile ride the same scale, so they do not underflow just because some other row of the matrix had a giant outlier.

Tensor-core accumulators are short. V3 periodically promotes partial sums to FP32 (every 128 along K) so a long reduction does not rot. Scales are online: computed from the current tile, not from a delayed running max that can lag a spike.

tile · s=0.5 · 448

MANIM Per-tile scale maps amax to E4M3 max 448.

Mantissa over extra exponent

FP8 comes in two common flavours. E5M2 has more range. E4M3 has more precision. V3 uses E4M3 everywhere in the GEMMs, including the backward, because the tile scale already supplies a shared exponent. You can spend the bits on mantissa.

OCP float8_e4m3fn has no infinities. The largest finite value is 448 (\(1.75 \times 2^{8}\)). That is not 240, which is the max of a different E4M3 convention. The check script builds the grid and asserts 448 is on it.

Do not quantise “to FP8” with an INT8 formula (scale by 127 and round to an integer). FP8 has an exponent. You scale so amax → 448, then round onto the E4M3 lattice.

A tile scaler and an E4M3 grid

This is not a Tensor-Core kernel. It is the arithmetic the kernel is approximating: per-tile scale, clip to 448, nearest grid point. Moments are tagged torch.bfloat16 so the fact cannot drift.

fp8.pyE4M3 max 448; 1×128 and 128×128 scales; moments BF16
"""DeepSeek-V3's FP8 training pieces, in miniature.

Generic number formats (IEEE 754, FP16 vs BF16, INT8) live in
Beyond the Forward Pass. This file is the DeepSeek recipe:

  * E4M3 (max finite value 448) for the GEMMs
  * a scale per 1×128 activation tile or 128×128 weight block
  * scales computed from the current tensor, not from a stale running max
  * optimizer moments stay BF16; master weights stay FP32
"""
import torch

# OCP float8_e4m3fn. Exponent 15 with mantissa 7 is NaN; 448 is the largest finite.
E4M3_MAX = 448.0
# AdamW first and second moments. DeepSeek-V3 stores these in BF16, not FP32.
MOMENT_DTYPE = torch.bfloat16
MASTER_DTYPE = torch.float32


def e4m3_grid():
    """Every non-negative finite E4M3fn value, including the extra exp=15 codes."""
    vals = [0.0]
    for exp in range(15):
        for mant in range(8):
            if exp == 0:
                # Subnormals: 2^{-6} * (mant / 8) = mant / 512.
                if mant:
                    vals.append(mant / 512.0)
            else:
                vals.append((1.0 + mant / 8.0) * (2.0 ** (exp - 7)))
    # Extra finites: exponent all-ones, mantissa 0..6. Max is 1.75 * 2^8 = 448.
    for mant in range(7):
        vals.append((1.0 + mant / 8.0) * (2.0 ** 8))
    return torch.tensor(vals, dtype=torch.float32)


# Built once. Used to round a real number onto the E4M3 lattice.
_GRID = None


def _grid():
    global _GRID
    if _GRID is None:
        _GRID = e4m3_grid()
    return _GRID


def round_to_e4m3(x):
    """Clip to ±448, then pick the nearest E4M3fn value (sign copied back)."""
    grid = _grid().to(device=x.device)
    mag = x.abs().clamp(max=E4M3_MAX)
    # Distances against the positive grid; restore sign after.
    idx = torch.abs(mag.unsqueeze(-1) - grid).argmin(dim=-1)
    return grid[idx] * x.sign().where(x != 0, torch.ones_like(x))


def tile_scale(tile, max_finite=E4M3_MAX):
    """One scale so this tile's largest magnitude maps to 448."""
    amax = tile.abs().max().clamp(min=1e-12)
    return amax / max_finite


def quantize_tile(tile, max_finite=E4M3_MAX):
    """Return (fp8-rounded tile, scale) reconstructed in FP32."""
    scale = tile_scale(tile, max_finite)
    q = round_to_e4m3(tile / scale)
    return q * scale, scale


def activation_scales(x, tile=128):
    """Per-token, per-128-channel scales. x is (tokens, channels)."""
    tokens, channels = x.shape
    assert channels % tile == 0
    grouped = x.view(tokens, channels // tile, tile)
    amax = grouped.abs().amax(dim=-1).clamp(min=1e-12)   # (tokens, n_tiles)
    return amax / E4M3_MAX


def weight_scales(w, block=128):
    """Per 128×128 block scales. w is (out, in)."""
    out, inn = w.shape
    assert out % block == 0 and inn % block == 0
    blocks = w.view(out // block, block, inn // block, block)
    amax = blocks.abs().amax(dim=(1, 3)).clamp(min=1e-12)  # (out_blocks, in_blocks)
    return amax / E4M3_MAX
  • E4M3_MAX = 448.0 is the number to remember. The grid builder exists so the check can see 448 is actually representable.
  • Activation scales are (tokens, channels/128). Weight scales are (out/128, in/128).
  • Online means amax is taken on the tile you have now, not on yesterday’s batch.

Mistakes learners make

“E4M3 max is 240.” That is another encoding. V3 uses the OCP fn variant. Max finite is 448.

“Optimizer state is FP32.” Master weights are FP32. AdamW moments are BF16.

Quantising FP8 like INT8. There is no “times 127”. Map amax to 448, then round to an E4M3 value.

One scale per tensor. An activation outlier in one token would crush every other token’s mantissa. The 1×128 tile is the fix.

What to remember

  • GEMMs in FP8 E4M3. Master weights FP32. Optimizer moments BF16.
  • Scales: 1×128 for activations, 128×128 for weights, computed online. amax / 448.
  • E4M3’s largest finite value is 448. Tile scales make the extra exponent of E5M2 less useful.
  • Promote partial sums often. Store backward activations in FP8 where you can.

Check yourself

1. What is the largest finite E4M3 value V3 uses?

448.

2. In which format are AdamW’s two moment tensors stored?

BF16. Not FP32.

3. A 256-channel activation row: how many scales?

Two. One per 1×128 tile.

4. Why not delayed scaling?

A spike in this step would be measured with last step’s amax, so you would clip or underflow. V3 computes the scale from the current tile.

Go to the source

Keep going