BEYOND THE FORWARD PASS · CHAPTER 03

Number Formats &
Quantization

Weights start as FP16 or BF16. Serving packs them into FP8, INT8 or INT4 so every decode step reads fewer bytes. The formats are not the same recipe.

~18 min read1-D toy tensorTested Python

What you'll learn

  • How FP16, BF16, FP8, INT8 and INT4 spend their bits
  • The symmetric absmax recipe for integers
  • Why FP8 E4M3’s max finite value is 448, not 127
  • What a five-number tensor looks like after INT8 and INT4
  • How those formats shrink Llama 3 8B on disk

Before you start

  • A linear layer is a matrix–vector product
  • Life of a prompt (decode is memory-bound)

Same shelf, coarser labels

A spice jar with a 100-mark scale and a jar with a 16-mark scale can hold the same cumin. The coarse jar just cannot tell 3.20 from 3.15. Quantization is that coarse jar: keep the range, spend fewer bits on the marks.

Integer formats (INT8, INT4) place those marks on a uniform grid. Floating formats (FP16, BF16, FP8) place them on an exponential grid, so small numbers keep relative precision. Mixing the two recipes is the usual source of wrong numbers.

Where the bits go

FormatSignExponentMantissa / payloadMax finite
FP321823≈ 3.4×10³⁸
FP16151065,504
BF16187≈ 3.4×10³⁸
FP8 E4M3143448
FP8 E5M215257,344
INT8uniform integers−128 … 127
INT4uniform integers−8 … 7
E4M3 spends mantissa over range: 3 fraction bits, max 448. E5M2 spends range over mantissa. DeepSeek-V3 trains GEMMs in E4M3; optimizer moments stay BF16.

BF16 keeps FP32’s exponent, so it does not overflow on the same large activations that overflow FP16. It just has a coarser fraction.

A five-number tensor

Own vector \(x = (1.60,\ -0.40,\ 3.20,\ 0.00,\ -2.40)\). The absmax is 3.20. Symmetric quantization:

scale is the absmax divided by the largest positive code. Round, clamp, store the code. Reconstruct with q times s.

INT8 uses \(q_{\mathrm{max}} = 127\), so \(s = 3.20 / 127 \approx 0.0252\).

\(x\)INT8 \(q\)Reconstructed
1.60631.587
−0.40−16−0.403
3.201273.200
0.0000.000
−2.40−95−2.394

INT4 uses \(q_{\mathrm{max}} = 7\), so \(s = 3.20 / 7 \approx 0.457\). Codes: 4, −1, 7, 0, −5. Reconstructions sit much further from \(x\) (1.829, −0.457, 3.200, 0, −2.286). Zero is still free in both formats: a stored 0 dequantizes to 0.

x · s · q

MANIM Scale by absmax/127, round, clamp. Zero stays free.

Scale to 448, then round on a float grid

E4M3’s largest finite value is 448. To put this tensor into E4M3 you pick a scale so 3.20 maps to 448, which is \(448 / 3.20 = 140\). Then you round each scaled number to the nearest E4M3 value — a number with an exponent and a 3-bit fraction, not an integer in −127 … 127.

INT8’s 127 and E4M3’s 448 are different kinds of maximum. One is the last integer code. The other is the last finite floating-point value of the format. Do not write round(x / s × 127) and call it FP8.

127 · 448 · warn

MANIM E4M3’s 448 is a float max, not an integer code.

Outliers still hurt, for a different reason than in INT8. A single huge activation eats the exponent range, and the rest of the tensor underflows to zero. That is why DeepSeek-V3 scales in 1×128 tiles instead of one scale per tensor, and why it keeps sensitive ops in higher precision.

Llama 3 8B on disk

8.03 billion parameters. Bytes = parameters × bits / 8.

FormatBitsWeights
FP16 / BF161615 GiB
FP8 or INT887.5 GiB
INT443.7 GiB

Decode is memory-bound, so halving the weight traffic often nearly doubles tokens per second, until the cache itself dominates. Cache quantization is a separate knob.

15 GiB · 7.5 · 3.7

MANIM 8.03B params: 15 → 7.5 → 3.7 GiB as bits drop.

The round-trip you can re-run

quantize.pyINT8 / INT4 round-trip, E4M3 max 448
"""Symmetric absmax quantization for a 1-D toy tensor.

INT8 and INT4 share the same recipe: pick a scale from the max absolute
value, round, clamp. FP8 E4M3 is different: after scaling you round to
the nearest value of a floating-point grid whose max finite value is 448.
"""

# Own 1-D tensor. Five numbers a reader can quantize by hand.
X = [1.60, -0.40, 3.20, 0.00, -2.40]


def absmax(xs):
    return max(abs(x) for x in xs)


def quantize_int(xs, n_levels_pos):
    """Symmetric absmax quantize onto integers in [-n_levels_pos, n_levels_pos].

    INT8 uses 127. INT4 uses 7. Zero is always free (scale * 0 = 0).
    """
    amax = absmax(xs)
    scale = amax / n_levels_pos
    q, recon = [], []
    for x in xs:
        code = round(x / scale)
        if code > n_levels_pos:
            code = n_levels_pos
        if code < -n_levels_pos:
            code = -n_levels_pos
        q.append(int(code))
        recon.append(code * scale)
    return scale, q, recon


def mse(xs, ys):
    return sum((a - b) ** 2 for a, b in zip(xs, ys)) / len(xs)


def int8_roundtrip(xs=None):
    xs = X if xs is None else xs
    return quantize_int(xs, 127)


def int4_roundtrip(xs=None):
    xs = X if xs is None else xs
    return quantize_int(xs, 7)


# OCP float8_e4m3fn: 4 exponent bits, 3 mantissa bits, no infinities.
# Exponent field 15 with mantissa 7 is NaN. The largest finite value is 448.
E4M3_MAX = 448.0


def fp8_e4m3_scale(xs=None):
    """Scale that maps the tensor's absmax onto the E4M3 max of 448."""
    xs = X if xs is None else xs
    return E4M3_MAX / absmax(xs)


def weight_bytes(n_params, bits):
    """Storage for n_params numbers at the given bit width."""
    return n_params * bits / 8

Mistakes learners make

Using 127 as the FP8 max. That is INT8. E4M3’s max finite value is 448. E5M2’s is 57,344.

“FP16 and BF16 are interchangeable.” Same size, different split. BF16 keeps range. FP16 keeps fraction. Overflow vs rounding error are different failures.

Forgetting the scale lives next to the tensor. The codes 63 and −16 mean nothing without \(s\). Per-channel or per-group scales are extra storage, still tiny next to the codes.

Quantizing the optimizer the same way. DeepSeek-V3 keeps Adam moments in BF16 even when GEMMs run in FP8.

What to remember

  • FP16 and BF16 split 16 bits differently. FP8 E4M3’s max is 448.
  • INT8 / INT4: scale = absmax / q_max, round, clamp. Reconstruct with q × s.
  • FP8 rounds onto a float grid after scaling to 448, not onto 127 integers.
  • Llama 3 8B: about 15 GiB in 16-bit, 7.5 GiB in 8-bit, 3.7 GiB in 4-bit.

Check yourself

1. What is the INT8 code for 3.20 on the toy tensor?

127. It is the absmax, so it saturates the positive range. Reconstruction is exact: 127 × (3.20/127) = 3.20.

2. Why is E4M3’s max 448, not 240?

OCP float8_e4m3fn uses exponent 15 with mantissa 0–6 as finite values. The largest is 1.75 × 2⁸ = 448. Mantissa 7 at that exponent is NaN.

3. Does INT4 store zero exactly on this tensor?

Yes. The code 0 reconstructs as 0 for any scale.

4. About how large is Llama 3 8B in INT4?

3.7 GiB of weights. The KV cache is extra and still usually bf16 unless you quantize it too.

Go to the source

Keep going