DECODING DEEPSEEK · CHAPTER 02
GQA shares keys. MLA compresses them. One latent per token, and every head still gets its own view.
In Deep Cuts 06 the cache stores a key card and a value card per KV head. MQA leaves one box of cards. GQA leaves a few. The readers (query heads) still ask different questions. They just share the cards.
MLA keeps a different kind of box: a compressed summary of the token, 512 numbers long. At decode time each head can stretch that summary back into its own key and value. The box is small. The views stay distinct.
Think of a coat-check ticket. You do not carry the coat. You carry a small tag. The attendant reconstructs the coat when you ask. MLA caches the tag.
Deep Cuts 06 ran this chatbot on Llama 3 8B: 32 users, 8,192 tokens each. GQA costs 32 GiB. Forcing MHA costs 128 GiB. Those two numbers stay. They are how the two series agree.
Now put the same load on a DeepSeek-V3-shaped stack: 61 layers, 128 heads, head size 128, bf16. Recap of the cache formula, one line: MHA stores \(2 \cdot L \cdot H \cdot d_h \cdot 2\) bytes per token.
two tensors (K and V), times layers, times heads, times head size, times two bytes.
MLA, in this chapter, caches only the 512-number latent:
one latent per layer, 61 layers, two bytes each. No separate K and V.
| Attention, V3-shaped | Per token | 32 users × 8,192 |
|---|---|---|
| MHA (128 KV heads) | 3.81 MiB | 976 GiB |
| GQA with 8 KV heads | 244 KiB | 61 GiB |
| MLA, latent 512 | 61 KiB | 15.25 GiB |
MHA · GQA · MLA
Interactive · The same 32 users
How does the V3-shaped cache grow with context, for MHA, GQA-8 and MLA?
Write \(d_c = 512\) for the latent size. For a token hidden state \(x\):
Compress. \(c_{KV} = \operatorname{RMSNorm}(x\,W_{DKV})\). This is the only vector that goes into the cache.
Up-project per head. \(k_h = c_{KV}\,W_{UK}^{(h)}\) and \(v_h = c_{KV}\,W_{UV}^{(h)}\). Heads disagree because the \(W\) matrices disagree, not because they stored different caches.
Queries are ordinary. \(q_h = x\,W_{Q}^{(h)}\). Queries are not cached, so they can stay fat. V3 also down-projects them (\(d_c' = 1536\)) to save activation memory in training. That does not change the cache.
one 512-vector per token per layer. Keys and values are views of this, not extra stored rows.
A tiny worked size, before the 61-layer jump. Two heads, head size 4, latent size 4. MHA stores \(2 \cdot 2 \cdot 4 = 16\) numbers per token. MLA stores 4. Same 4× gap you will see at full size as 64×, because V3 has 128 heads of size 128.
down · cache · up
If you reconstruct \(k_h\) for every cached token, you have paid the cache in compute. The absorption trick folds the up-projection into the query instead.
the score is a dot product between a d_c-vector (the absorbed query) and the cached latent. Full keys never appear.
Every head still has its own absorbed query, because \(W_Q^{(h)}\) and \(W_{UK}^{(h)}\) are per head. The cached thing they all read is the same \(c\). At decode time MLA is multi-query attention over a 512-d key, with the diversity living in the absorbed projections.
Values can be absorbed the same way: fold \(W_{UV}\) into \(W_O\) so you mix the latent directly into the output. The code below materialises values so the check can compare against a normal forward pass. The algebra is the same idea.
Absorption is an inference rewrite. Training still computes keys and values. The next chapter is about why a position-dependent rotation between \(W_Q\) and \(W_{UK}\) would make this rewrite illegal.
This is the no-RoPE core. Prefill and decode share forward. absorbed_decode is the one-token path that never forms \(K\). The check script confirms both match a full causal pass.
"""Multi-head latent attention, without RoPE.
Instead of caching a key and a value per head, we cache one small latent
per token. Heads reconstruct their own keys and values from that latent.
At decode time we can fold the key up-projection into the query, so the
full keys never have to exist in memory.
RoPE needs a second cached vector. That is the next chapter.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
class LatentCache:
"""Compressed KV latents for one layer. Shape (batch, tokens, d_c)."""
def __init__(self):
self.c = None
def append(self, c_new):
# Grow along the token axis. Same idea as a KV cache, smaller payload.
if self.c is None:
self.c = c_new
else:
self.c = torch.cat([self.c, c_new], dim=1)
return self.c
class MultiHeadLatentAttention(nn.Module):
"""Low-rank KV compression. Queries stay full-size; keys and values do not."""
def __init__(self, d_model, n_heads, d_head, d_c):
super().__init__()
self.n_heads = n_heads
self.d_head = d_head
self.d_c = d_c
# One shared down-projection. This is what we cache.
self.w_dkv = nn.Linear(d_model, d_c, bias=False)
# Per-head up-projections from the latent to keys and values.
self.w_uk = nn.Linear(d_c, n_heads * d_head, bias=False)
self.w_uv = nn.Linear(d_c, n_heads * d_head, bias=False)
# Queries still come from the full hidden state.
self.w_q = nn.Linear(d_model, n_heads * d_head, bias=False)
self.w_o = nn.Linear(n_heads * d_head, d_model, bias=False)
# DeepSeek-V3 puts RMSNorm on the latent before it is stored.
self.rms = nn.RMSNorm(d_c)
def latent(self, x):
return self.rms(self.w_dkv(x))
def _heads(self, tensor, tokens):
# (B, T, H * d_h) -> (B, H, T, d_h)
B = tensor.shape[0]
return tensor.view(B, tokens, self.n_heads, self.d_head).transpose(1, 2)
def forward(self, x, cache=None):
"""x: (B, T_new, d_model). With a cache, attend over every token so far."""
B, T_new, _ = x.shape
c_new = self.latent(x)
c = cache.append(c_new) if cache is not None else c_new
T_all = c.shape[1]
q = self._heads(self.w_q(x), T_new)
k = self._heads(self.w_uk(c), T_all)
v = self._heads(self.w_uv(c), T_all)
scale = self.d_head ** 0.5
scores = q @ k.transpose(-2, -1) / scale
# New queries sit at the end of the cache, same as ordinary decode.
pos_q = torch.arange(T_all - T_new, T_all, device=x.device).unsqueeze(1)
pos_k = torch.arange(T_all, device=x.device).unsqueeze(0)
scores = scores.masked_fill(pos_k > pos_q, float("-inf"))
ctx = torch.softmax(scores, dim=-1) @ v
ctx = ctx.transpose(1, 2).contiguous().view(B, T_new, -1)
return self.w_o(ctx)
def absorbed_decode(self, x, cache):
"""One new token. Fold W_UK into the query so keys stay in latent space.
Math: score_h = (x W_Qh) (c W_UKh)^T = (x W_Qh W_UKh^T) c^T.
The thing in parentheses is a d_c-vector. Every head still has its
own vector, but they all read the same cached latent.
"""
B = x.shape[0]
c = cache.append(self.latent(x)) # (B, T_all, d_c)
w_q = self.w_q.weight.view(self.n_heads, self.d_head, -1)
w_uk = self.w_uk.weight.view(self.n_heads, self.d_head, self.d_c)
# q: (B, 1, H, d_h) q_abs: (B, 1, H, d_c)
q = torch.einsum("btd,hid->bthi", x, w_q)
q_abs = torch.einsum("bthi,hic->bthc", q, w_uk)
scores = torch.einsum("bthc,btc->bht", q_abs, c) / (self.d_head ** 0.5)
weights = torch.softmax(scores, dim=-1)
v = self._heads(self.w_uv(c), c.shape[1])
ctx = torch.einsum("bht,bhtd->bhd", weights, v).reshape(B, 1, -1)
return self.w_o(ctx)
LatentCache grows on dim 1, the token axis. It stores \(c\), not \(K\) and \(V\).einsum in absorbed_decode is \(q_{\text{abs}} = x\,W_Q^{(h)}(W_{UK}^{(h)})^{\top}\), one \(d_c\)-vector per head.“MLA is GQA with a funny name.” GQA stores 8 (or so) full-size KV heads. MLA stores one 512-d latent. V3 does not use GQA.
“The latent is 576.” 576 = 512 + 64. The 64 is a RoPE key, added in the next chapter. This chapter’s cache is 512 numbers, 61 KiB per token.
Absorbing by splitting \(x\) into head-sized chunks. The absorbed query uses the full \(x\) and the per-head \(W_Q^{(h)}\). You do not slice the input in half and call that a head.
“Absorption is used in training.” Training materialises keys and values. Absorption is a decode rewrite.
One latent reconstructs both keys and values. You do not store a separate value latent.
976 GiB and 15.25 GiB.
They share the cached latent. Each head has its own absorbed query \(x W_Q^{(h)} (W_{UK}^{(h)})^{\top}\).
No. 61 KiB is \(512 \times 61 \times 2\). The 64-d RoPE key lands in the next chapter, and the total becomes 70,272 bytes.