BEYOND THE FORWARD PASS · CHAPTER 03
Weights start as FP16 or BF16. Serving packs them into FP8, INT8 or INT4 so every decode step reads fewer bytes. The formats are not the same recipe.
A spice jar with a 100-mark scale and a jar with a 16-mark scale can hold the same cumin. The coarse jar just cannot tell 3.20 from 3.15. Quantization is that coarse jar: keep the range, spend fewer bits on the marks.
Integer formats (INT8, INT4) place those marks on a uniform grid. Floating formats (FP16, BF16, FP8) place them on an exponential grid, so small numbers keep relative precision. Mixing the two recipes is the usual source of wrong numbers.
| Format | Sign | Exponent | Mantissa / payload | Max finite |
|---|---|---|---|---|
| FP32 | 1 | 8 | 23 | ≈ 3.4×10³⁸ |
| FP16 | 1 | 5 | 10 | 65,504 |
| BF16 | 1 | 8 | 7 | ≈ 3.4×10³⁸ |
| FP8 E4M3 | 1 | 4 | 3 | 448 |
| FP8 E5M2 | 1 | 5 | 2 | 57,344 |
| INT8 | uniform integers | −128 … 127 | ||
| INT4 | uniform integers | −8 … 7 | ||
BF16 keeps FP32’s exponent, so it does not overflow on the same large activations that overflow FP16. It just has a coarser fraction.
Own vector \(x = (1.60,\ -0.40,\ 3.20,\ 0.00,\ -2.40)\). The absmax is 3.20. Symmetric quantization:
scale is the absmax divided by the largest positive code. Round, clamp, store the code. Reconstruct with q times s.
INT8 uses \(q_{\mathrm{max}} = 127\), so \(s = 3.20 / 127 \approx 0.0252\).
| \(x\) | INT8 \(q\) | Reconstructed |
|---|---|---|
| 1.60 | 63 | 1.587 |
| −0.40 | −16 | −0.403 |
| 3.20 | 127 | 3.200 |
| 0.00 | 0 | 0.000 |
| −2.40 | −95 | −2.394 |
INT4 uses \(q_{\mathrm{max}} = 7\), so \(s = 3.20 / 7 \approx 0.457\). Codes: 4, −1, 7, 0, −5. Reconstructions sit much further from \(x\) (1.829, −0.457, 3.200, 0, −2.286). Zero is still free in both formats: a stored 0 dequantizes to 0.
x · s · q
E4M3’s largest finite value is 448. To put this tensor into E4M3 you pick a scale so 3.20 maps to 448, which is \(448 / 3.20 = 140\). Then you round each scaled number to the nearest E4M3 value — a number with an exponent and a 3-bit fraction, not an integer in −127 … 127.
INT8’s 127 and E4M3’s 448 are different kinds of maximum. One is the last integer code. The other is the last finite floating-point value of the format. Do not write round(x / s × 127) and call it FP8.
127 · 448 · warn
Outliers still hurt, for a different reason than in INT8. A single huge activation eats the exponent range, and the rest of the tensor underflows to zero. That is why DeepSeek-V3 scales in 1×128 tiles instead of one scale per tensor, and why it keeps sensitive ops in higher precision.
8.03 billion parameters. Bytes = parameters × bits / 8.
| Format | Bits | Weights |
|---|---|---|
| FP16 / BF16 | 16 | 15 GiB |
| FP8 or INT8 | 8 | 7.5 GiB |
| INT4 | 4 | 3.7 GiB |
Decode is memory-bound, so halving the weight traffic often nearly doubles tokens per second, until the cache itself dominates. Cache quantization is a separate knob.
15 GiB · 7.5 · 3.7
"""Symmetric absmax quantization for a 1-D toy tensor. INT8 and INT4 share the same recipe: pick a scale from the max absolute value, round, clamp. FP8 E4M3 is different: after scaling you round to the nearest value of a floating-point grid whose max finite value is 448. """ # Own 1-D tensor. Five numbers a reader can quantize by hand. X = [1.60, -0.40, 3.20, 0.00, -2.40] def absmax(xs): return max(abs(x) for x in xs) def quantize_int(xs, n_levels_pos): """Symmetric absmax quantize onto integers in [-n_levels_pos, n_levels_pos]. INT8 uses 127. INT4 uses 7. Zero is always free (scale * 0 = 0). """ amax = absmax(xs) scale = amax / n_levels_pos q, recon = [], [] for x in xs: code = round(x / scale) if code > n_levels_pos: code = n_levels_pos if code < -n_levels_pos: code = -n_levels_pos q.append(int(code)) recon.append(code * scale) return scale, q, recon def mse(xs, ys): return sum((a - b) ** 2 for a, b in zip(xs, ys)) / len(xs) def int8_roundtrip(xs=None): xs = X if xs is None else xs return quantize_int(xs, 127) def int4_roundtrip(xs=None): xs = X if xs is None else xs return quantize_int(xs, 7) # OCP float8_e4m3fn: 4 exponent bits, 3 mantissa bits, no infinities. # Exponent field 15 with mantissa 7 is NaN. The largest finite value is 448. E4M3_MAX = 448.0 def fp8_e4m3_scale(xs=None): """Scale that maps the tensor's absmax onto the E4M3 max of 448.""" xs = X if xs is None else xs return E4M3_MAX / absmax(xs) def weight_bytes(n_params, bits): """Storage for n_params numbers at the given bit width.""" return n_params * bits / 8
Using 127 as the FP8 max. That is INT8. E4M3’s max finite value is 448. E5M2’s is 57,344.
“FP16 and BF16 are interchangeable.” Same size, different split. BF16 keeps range. FP16 keeps fraction. Overflow vs rounding error are different failures.
Forgetting the scale lives next to the tensor. The codes 63 and −16 mean nothing without \(s\). Per-channel or per-group scales are extra storage, still tiny next to the codes.
Quantizing the optimizer the same way. DeepSeek-V3 keeps Adam moments in BF16 even when GEMMs run in FP8.
127. It is the absmax, so it saturates the positive range. Reconstruction is exact: 127 × (3.20/127) = 3.20.
OCP float8_e4m3fn uses exponent 15 with mantissa 0–6 as finite values. The largest is 1.75 × 2⁸ = 448. Mantissa 7 at that exponent is NaN.
Yes. The code 0 reconstructs as 0 for any scale.
3.7 GiB of weights. The KV cache is extra and still usually bf16 unless you quantize it too.