DECODING DEEPSEEK · CHAPTER 02

Multi-Head Latent Attention

GQA shares keys. MLA compresses them. One latent per token, and every head still gets its own view.

~20 min read1 interactiveTested PyTorch code

What you'll learn

  • Why shrinking KV heads is not the only way to shrink the cache
  • How a 512-number latent replaces per-head keys and values
  • How to fold the key up-projection into the query at decode time
  • The 32-user chatbot bill for MHA, GQA and MLA, using the same numbers as Deep Cuts 06

Before you start

One filing cabinet, many readers

In Deep Cuts 06 the cache stores a key card and a value card per KV head. MQA leaves one box of cards. GQA leaves a few. The readers (query heads) still ask different questions. They just share the cards.

MLA keeps a different kind of box: a compressed summary of the token, 512 numbers long. At decode time each head can stretch that summary back into its own key and value. The box is small. The views stay distinct.

Think of a coat-check ticket. You do not carry the coat. You carry a small tag. The attendant reconstructs the coat when you ask. MLA caches the tag.

32 users, 8,192 tokens, three attention styles

Deep Cuts 06 ran this chatbot on Llama 3 8B: 32 users, 8,192 tokens each. GQA costs 32 GiB. Forcing MHA costs 128 GiB. Those two numbers stay. They are how the two series agree.

Now put the same load on a DeepSeek-V3-shaped stack: 61 layers, 128 heads, head size 128, bf16. Recap of the cache formula, one line: MHA stores \(2 \cdot L \cdot H \cdot d_h \cdot 2\) bytes per token.

two tensors (K and V), times layers, times heads, times head size, times two bytes.

MLA, in this chapter, caches only the 512-number latent:

one latent per layer, 61 layers, two bytes each. No separate K and V.

Attention, V3-shapedPer token32 users × 8,192
MHA (128 KV heads)3.81 MiB976 GiB
GQA with 8 KV heads244 KiB61 GiB
MLA, latent 51261 KiB15.25 GiB
Same chatbot as Deep Cuts 06. MLA is 64× smaller than MHA on this stack (\(2 \cdot 128 \cdot 128 / 512\)). Chapter 03 adds 64 RoPE numbers and the bill becomes 70,272 bytes, still far under GQA.

MHA · GQA · MLA

MANIM Same chatbot: MLA cache is 15.25 GiB vs MHA 976 GiB.

Interactive · The same 32 users

How does the V3-shaped cache grow with context, for MHA, GQA-8 and MLA?

Down, cache, up

Write \(d_c = 512\) for the latent size. For a token hidden state \(x\):

  1. Compress. \(c_{KV} = \operatorname{RMSNorm}(x\,W_{DKV})\). This is the only vector that goes into the cache.

  2. Up-project per head. \(k_h = c_{KV}\,W_{UK}^{(h)}\) and \(v_h = c_{KV}\,W_{UV}^{(h)}\). Heads disagree because the \(W\) matrices disagree, not because they stored different caches.

  3. Queries are ordinary. \(q_h = x\,W_{Q}^{(h)}\). Queries are not cached, so they can stay fat. V3 also down-projects them (\(d_c' = 1536\)) to save activation memory in training. That does not change the cache.

one 512-vector per token per layer. Keys and values are views of this, not extra stored rows.

A tiny worked size, before the 61-layer jump. Two heads, head size 4, latent size 4. MHA stores \(2 \cdot 2 \cdot 4 = 16\) numbers per token. MLA stores 4. Same 4× gap you will see at full size as 64×, because V3 has 128 heads of size 128.

down · cache · up

MANIM Only c_KV is cached; heads up-project from the same latent.

Never build the keys at decode time

If you reconstruct \(k_h\) for every cached token, you have paid the cache in compute. The absorption trick folds the up-projection into the query instead.

the score is a dot product between a d_c-vector (the absorbed query) and the cached latent. Full keys never appear.

Every head still has its own absorbed query, because \(W_Q^{(h)}\) and \(W_{UK}^{(h)}\) are per head. The cached thing they all read is the same \(c\). At decode time MLA is multi-query attention over a 512-d key, with the diversity living in the absorbed projections.

Values can be absorbed the same way: fold \(W_{UV}\) into \(W_O\) so you mix the latent directly into the output. The code below materialises values so the check can compare against a normal forward pass. The algebra is the same idea.

Absorption is an inference rewrite. Training still computes keys and values. The next chapter is about why a position-dependent rotation between \(W_Q\) and \(W_{UK}\) would make this rewrite illegal.

A ropeless MLA you can run

This is the no-RoPE core. Prefill and decode share forward. absorbed_decode is the one-token path that never forms \(K\). The check script confirms both match a full causal pass.

mla.pycached decode = full pass; absorbed last token = full pass
"""Multi-head latent attention, without RoPE.

Instead of caching a key and a value per head, we cache one small latent
per token. Heads reconstruct their own keys and values from that latent.
At decode time we can fold the key up-projection into the query, so the
full keys never have to exist in memory.

RoPE needs a second cached vector. That is the next chapter.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F


class LatentCache:
    """Compressed KV latents for one layer. Shape (batch, tokens, d_c)."""

    def __init__(self):
        self.c = None

    def append(self, c_new):
        # Grow along the token axis. Same idea as a KV cache, smaller payload.
        if self.c is None:
            self.c = c_new
        else:
            self.c = torch.cat([self.c, c_new], dim=1)
        return self.c


class MultiHeadLatentAttention(nn.Module):
    """Low-rank KV compression. Queries stay full-size; keys and values do not."""

    def __init__(self, d_model, n_heads, d_head, d_c):
        super().__init__()
        self.n_heads = n_heads
        self.d_head = d_head
        self.d_c = d_c
        # One shared down-projection. This is what we cache.
        self.w_dkv = nn.Linear(d_model, d_c, bias=False)
        # Per-head up-projections from the latent to keys and values.
        self.w_uk = nn.Linear(d_c, n_heads * d_head, bias=False)
        self.w_uv = nn.Linear(d_c, n_heads * d_head, bias=False)
        # Queries still come from the full hidden state.
        self.w_q = nn.Linear(d_model, n_heads * d_head, bias=False)
        self.w_o = nn.Linear(n_heads * d_head, d_model, bias=False)
        # DeepSeek-V3 puts RMSNorm on the latent before it is stored.
        self.rms = nn.RMSNorm(d_c)

    def latent(self, x):
        return self.rms(self.w_dkv(x))

    def _heads(self, tensor, tokens):
        # (B, T, H * d_h) -> (B, H, T, d_h)
        B = tensor.shape[0]
        return tensor.view(B, tokens, self.n_heads, self.d_head).transpose(1, 2)

    def forward(self, x, cache=None):
        """x: (B, T_new, d_model). With a cache, attend over every token so far."""
        B, T_new, _ = x.shape
        c_new = self.latent(x)
        c = cache.append(c_new) if cache is not None else c_new
        T_all = c.shape[1]

        q = self._heads(self.w_q(x), T_new)
        k = self._heads(self.w_uk(c), T_all)
        v = self._heads(self.w_uv(c), T_all)

        scale = self.d_head ** 0.5
        scores = q @ k.transpose(-2, -1) / scale
        # New queries sit at the end of the cache, same as ordinary decode.
        pos_q = torch.arange(T_all - T_new, T_all, device=x.device).unsqueeze(1)
        pos_k = torch.arange(T_all, device=x.device).unsqueeze(0)
        scores = scores.masked_fill(pos_k > pos_q, float("-inf"))
        ctx = torch.softmax(scores, dim=-1) @ v
        ctx = ctx.transpose(1, 2).contiguous().view(B, T_new, -1)
        return self.w_o(ctx)

    def absorbed_decode(self, x, cache):
        """One new token. Fold W_UK into the query so keys stay in latent space.

        Math: score_h = (x W_Qh) (c W_UKh)^T = (x W_Qh W_UKh^T) c^T.
        The thing in parentheses is a d_c-vector. Every head still has its
        own vector, but they all read the same cached latent.
        """
        B = x.shape[0]
        c = cache.append(self.latent(x))              # (B, T_all, d_c)
        w_q = self.w_q.weight.view(self.n_heads, self.d_head, -1)
        w_uk = self.w_uk.weight.view(self.n_heads, self.d_head, self.d_c)
        # q: (B, 1, H, d_h)   q_abs: (B, 1, H, d_c)
        q = torch.einsum("btd,hid->bthi", x, w_q)
        q_abs = torch.einsum("bthi,hic->bthc", q, w_uk)
        scores = torch.einsum("bthc,btc->bht", q_abs, c) / (self.d_head ** 0.5)
        weights = torch.softmax(scores, dim=-1)
        v = self._heads(self.w_uv(c), c.shape[1])
        ctx = torch.einsum("bht,bhtd->bhd", weights, v).reshape(B, 1, -1)
        return self.w_o(ctx)
  • LatentCache grows on dim 1, the token axis. It stores \(c\), not \(K\) and \(V\).
  • einsum in absorbed_decode is \(q_{\text{abs}} = x\,W_Q^{(h)}(W_{UK}^{(h)})^{\top}\), one \(d_c\)-vector per head.
  • RMSNorm on the latent is what V3 does before the cache write.

Mistakes learners make

“MLA is GQA with a funny name.” GQA stores 8 (or so) full-size KV heads. MLA stores one 512-d latent. V3 does not use GQA.

“The latent is 576.” 576 = 512 + 64. The 64 is a RoPE key, added in the next chapter. This chapter’s cache is 512 numbers, 61 KiB per token.

Absorbing by splitting \(x\) into head-sized chunks. The absorbed query uses the full \(x\) and the per-head \(W_Q^{(h)}\). You do not slice the input in half and call that a head.

“Absorption is used in training.” Training materialises keys and values. Absorption is a decode rewrite.

What to remember

  • MLA caches \(c_{KV} \in \mathbb{R}^{512}\) per token per layer, then up-projects to per-head K and V.
  • On a V3-shaped stack the 32 × 8,192 chatbot is 15.25 GiB (MLA) vs 61 GiB (GQA-8) vs 976 GiB (MHA). Llama 3 8B GQA on the same load is still 32 GiB.
  • At decode time, fold \(W_{UK}\) into the query. Heads stay diverse. The cache stays one latent.
  • RoPE is not in this module yet. Adding it without care would break absorption. That is chapter 03, which brings the cache to 576 numbers.

Check yourself

1. Why is the MLA cache not \(2 \times 512\)?

One latent reconstructs both keys and values. You do not store a separate value latent.

2. The 32-user, 8,192-token chatbot on Llama 3 8B GQA is 32 GiB. What is it on V3-shaped MHA, and on MLA-512?

976 GiB and 15.25 GiB.

3. After absorption, what do all heads share, and what stays per-head?

They share the cached latent. Each head has its own absorbed query \(x W_Q^{(h)} (W_{UK}^{(h)})^{\top}\).

4. Does this chapter’s 61 KiB include RoPE?

No. 61 KiB is \(512 \times 61 \times 2\). The 64-d RoPE key lands in the next chapter, and the total becomes 70,272 bytes.

Go to the source

Keep going