DECODING DEEPSEEK · CHAPTER 03
A position-dependent rotation between \(W_Q\) and \(W_{UK}\) makes absorption illegal. DeepSeek splits the key into a content latent and a 64-number RoPE key.
RoPE rotates a vector by an angle that depends on its position. Nearby tokens get similar angles. The query-key score then depends on relative distance. That is the whole recap.
Absorption needs \(W_Q\) and \(W_{UK}\) to be fixed matrices. A rotation \(R_t\) that depends on which token you are looking at is not a fixed matrix. You cannot fold it in once and reuse it for every cached position.
Leave the filing-cabinet summaries (the 512-latent) un-rotated, so they still absorb. Keep a second, tiny key that is rotated, shared across heads, and cached next to the latent.
Last chapter, the score was
two fixed matrices, then a dot with the cached latent. Legal to precompute \(W_Q W_{UK}^{\top}\).
If you rotate the reconstructed key, the score becomes \(x W_Q^{(h)}\,R_t\,(W_{UK}^{(h)})^{\top} c^{\top}\). \(R_t\) is different for every cached token \(t\). There is no single absorbed matrix that works for the whole cache. You would have to rebuild keys every step, and the memory win vanishes.
Rotating only the query does not help. RoPE’s relative-position identity needs the same rotation family on both sides: \(\langle R_m q, R_n k\rangle = \langle q, R_{n-m} k\rangle\).
Split each query and key into a content half and a RoPE half, then concatenate. Values stay on the content path. They have no RoPE slice.
the dot product of concatenations is the sum of the two dots. No cross terms. No extra orthogonality assumption.
A 4-number check you can do by hand. \(q^{C}=[1,2]\), \(q^{R}=[3,4]\), \(k^{C}=[0.5,0.5]\), \(k^{R}=[1,0]\).
content · RoPE · 4.5
The two paths in V3:
\(c_{KV}\) is 512 numbers, RMSNorm’d, cached. \(k^{C}\) and \(v\) are up-projected from it. \(q^{C}\) is per head. This path never sees RoPE, so last chapter’s absorption still goes through.
\(k^{R} = R_t(x W_{KR})\) is 64 numbers, one per token, cached after rotation. \(q^{R}\) is per head, rotated the same way. Heads share the key. They do not share the query.
Why share \(k^{R}\) but not \(q^{R}\)? The cache only stores keys. One 64-vector is cheap. Queries are built for the new token only, so making them per-head costs no cache.
There is no \(v^{R}\). Values are a mix of content. Position already entered through the scores.
V3 down-projects queries through a 1536-d latent before splitting into \(q^{C}\) and \(q^{R}\). That saves training activation memory. It does not change what is cached. Long context on the RoPE slice uses YaRN; the content latent does not need it.
latent plus RoPE key, times 61 layers, times two bytes for bf16.
Scale factor against MHA on this stack: \(2 \cdot 128 \cdot 128 / 576 = 4N/9 \approx 56.9\times\).
Same 32-user, 8,192-token chatbot as Deep Cuts 06 and chapter 02:
| What you cache | 32 × 8,192 |
|---|---|
| V3-shaped MHA | 976 GiB |
| MLA, latent only (ch 02) | 15.25 GiB |
| MLA + 64-d RoPE key | 17.16 GiB |
Per decode step you append one 512-vector and one already-rotated 64-vector. You do not re-rotate the cache.
512+64 · 70,272 · hold
Notice w_kr maps to d_rope, not to n_heads * d_rope. The shared key is expanded across heads only when scoring. Rotate before the cache write. The check script confirms a prefill-then-decode run matches a full pass, and that the RoPE cache is (batch, tokens, d_rope).
"""MLA with decoupled RoPE.
RoPE is a position-dependent rotation. If you rotate the reconstructed
key, that rotation sits between W_Q and W_UK and you can no longer fold
them together. DeepSeek's fix: keep a RoPE-free content path (absorbable,
cached as c_KV) and a small shared RoPE key (cached as k_R).
Per token we store d_c + d_rope numbers. For DeepSeek-V3 that is 512 + 64.
"""
import torch
import torch.nn as nn
def rotate_pairs(x, cos, sin):
"""Rotate consecutive pairs (x0, x1), (x2, x3), ... by the given angles."""
x_even, x_odd = x[..., 0::2], x[..., 1::2]
rot_even = x_even * cos - x_odd * sin
rot_odd = x_even * sin + x_odd * cos
return torch.stack((rot_even, rot_odd), dim=-1).flatten(-2)
def rope_angles(seq_len, dim, base=10_000.0, device=None):
"""cos/sin for positions 0..seq_len-1, one pair of angles per two dims."""
# Even-odd pairing uses dim/2 frequencies.
freqs = 1.0 / (base ** (torch.arange(0, dim, 2, device=device).float() / dim))
pos = torch.arange(seq_len, device=device).float()
angles = torch.outer(pos, freqs) # (T, dim/2)
return torch.cos(angles), torch.sin(angles)
class DualCache:
"""Content latent and the shared RoPE key, for one layer."""
def __init__(self):
self.c = None # (batch, tokens, d_c)
self.k_r = None # (batch, tokens, d_rope), already rotated
def append(self, c_new, k_r_new):
if self.c is None:
self.c, self.k_r = c_new, k_r_new
else:
self.c = torch.cat([self.c, c_new], dim=1)
self.k_r = torch.cat([self.k_r, k_r_new], dim=1)
return self.c, self.k_r
class DecoupledMLA(nn.Module):
"""Content path through a latent; position path is a shared RoPE key."""
def __init__(self, d_model, n_heads, d_head, d_c, d_rope):
super().__init__()
self.n_heads = n_heads
self.d_head = d_head
self.d_c = d_c
self.d_rope = d_rope
# Content KV: compress, then up-project. RoPE never touches these.
self.w_dkv = nn.Linear(d_model, d_c, bias=False)
self.w_uk = nn.Linear(d_c, n_heads * d_head, bias=False)
self.w_uv = nn.Linear(d_c, n_heads * d_head, bias=False)
self.rms_kv = nn.RMSNorm(d_c)
# Shared RoPE key. One vector per token, not one per head.
self.w_kr = nn.Linear(d_model, d_rope, bias=False)
# Queries: content (per head) plus a RoPE slice (also per head).
self.w_q = nn.Linear(d_model, n_heads * d_head, bias=False)
self.w_qr = nn.Linear(d_model, n_heads * d_rope, bias=False)
self.w_o = nn.Linear(n_heads * d_head, d_model, bias=False)
def _split_heads(self, tensor, tokens, width):
B = tensor.shape[0]
return tensor.view(B, tokens, self.n_heads, width).transpose(1, 2)
def forward(self, x, cache=None, start_pos=0):
"""x: (B, T_new, d_model). start_pos is the first new token's index."""
B, T_new, _ = x.shape
c_new = self.rms_kv(self.w_dkv(x))
# Rotate the shared key by this token's position, then cache it.
cos, sin = rope_angles(start_pos + T_new, self.d_rope, device=x.device)
k_r_raw = self.w_kr(x) # (B, T_new, d_rope)
k_r_new = rotate_pairs(k_r_raw, cos[start_pos:], sin[start_pos:])
if cache is not None:
c, k_r = cache.append(c_new, k_r_new)
else:
c, k_r = c_new, k_r_new
T_all = c.shape[1]
q_c = self._split_heads(self.w_q(x), T_new, self.d_head)
k_c = self._split_heads(self.w_uk(c), T_all, self.d_head)
v = self._split_heads(self.w_uv(c), T_all, self.d_head)
q_r = self._split_heads(self.w_qr(x), T_new, self.d_rope)
q_r = rotate_pairs(q_r, cos[start_pos:], sin[start_pos:])
# Broadcast the shared RoPE key across heads. It is stored once.
k_r_heads = k_r.unsqueeze(1).expand(-1, self.n_heads, -1, -1)
# Concat per head: score = content_dot + rope_dot. No extra cross terms.
q = torch.cat([q_c, q_r], dim=-1)
k = torch.cat([k_c, k_r_heads], dim=-1)
scale = (self.d_head + self.d_rope) ** 0.5
scores = q @ k.transpose(-2, -1) / scale
pos_q = torch.arange(T_all - T_new, T_all, device=x.device).unsqueeze(1)
pos_k = torch.arange(T_all, device=x.device).unsqueeze(0)
scores = scores.masked_fill(pos_k > pos_q, float("-inf"))
ctx = torch.softmax(scores, dim=-1) @ v
ctx = ctx.transpose(1, 2).contiguous().view(B, T_new, -1)
return self.w_o(ctx)
rotate_pairs is even/odd RoPE. Apply it to the new slice only, using start_pos.Re-rotating cached \(k_R\) at the current position. Each key is rotated by its position, then stored. Re-applying the newest angle would smash relative position.
One attention over a concatenated mega-head. Concatenate per head, then run \(H\) softmaxes. A single 16-d softmax is not multi-head MLA.
“The two subspaces must be orthogonal.” Concatenation already splits the dot product. You do not need \(q^{C}\!\cdot k^{R} = 0\).
A per-head RoPE key in the cache. That would be \(128 \times 64\) extra numbers and would undo the win. V3 stores one 64-vector.
The rotation depends on the cached token's position, so there is no single matrix \(W_Q W_{UK}^{\top}\) that works for every key.
64. One shared vector, not 128 heads × 64.
4.5. Sum of 1.5 and 3, which matches the concatenated dot product.
\((512+64) \times 61 \times 2\) bytes: the bf16 cache for one token across V3.