DECODING DEEPSEEK · CHAPTER 03

MLA Meets RoPE

A position-dependent rotation between \(W_Q\) and \(W_{UK}\) makes absorption illegal. DeepSeek splits the key into a content latent and a 64-number RoPE key.

~18 min readTested PyTorch code576 numbers

What you'll learn

  • Why RoPE cannot sit inside the absorbed content path
  • How a shared RoPE key plus per-head RoPE queries restore position
  • Why the score is just content-dot plus RoPE-dot
  • The real V3 cache: 512 + 64 = 576 numbers, 70,272 bytes per token

Before you start

A lock that changes every step

RoPE rotates a vector by an angle that depends on its position. Nearby tokens get similar angles. The query-key score then depends on relative distance. That is the whole recap.

Absorption needs \(W_Q\) and \(W_{UK}\) to be fixed matrices. A rotation \(R_t\) that depends on which token you are looking at is not a fixed matrix. You cannot fold it in once and reuse it for every cached position.

Leave the filing-cabinet summaries (the 512-latent) un-rotated, so they still absorb. Keep a second, tiny key that is rotated, shared across heads, and cached next to the latent.

The rotation sits in the wrong place

Last chapter, the score was

two fixed matrices, then a dot with the cached latent. Legal to precompute \(W_Q W_{UK}^{\top}\).

If you rotate the reconstructed key, the score becomes \(x W_Q^{(h)}\,R_t\,(W_{UK}^{(h)})^{\top} c^{\top}\). \(R_t\) is different for every cached token \(t\). There is no single absorbed matrix that works for the whole cache. You would have to rebuild keys every step, and the memory win vanishes.

Rotating only the query does not help. RoPE’s relative-position identity needs the same rotation family on both sides: \(\langle R_m q, R_n k\rangle = \langle q, R_{n-m} k\rangle\).

Two paths, one score

Split each query and key into a content half and a RoPE half, then concatenate. Values stay on the content path. They have no RoPE slice.

the dot product of concatenations is the sum of the two dots. No cross terms. No extra orthogonality assumption.

A 4-number check you can do by hand. \(q^{C}=[1,2]\), \(q^{R}=[3,4]\), \(k^{C}=[0.5,0.5]\), \(k^{R}=[1,0]\).

content · RoPE · 4.5

MANIM Content and RoPE dots add — concat identity.

The two paths in V3:

Content (absorbable)

\(c_{KV}\) is 512 numbers, RMSNorm’d, cached. \(k^{C}\) and \(v\) are up-projected from it. \(q^{C}\) is per head. This path never sees RoPE, so last chapter’s absorption still goes through.

Position (shared key)

\(k^{R} = R_t(x W_{KR})\) is 64 numbers, one per token, cached after rotation. \(q^{R}\) is per head, rotated the same way. Heads share the key. They do not share the query.

Why share \(k^{R}\) but not \(q^{R}\)? The cache only stores keys. One 64-vector is cheap. Queries are built for the new token only, so making them per-head costs no cache.

There is no \(v^{R}\). Values are a mix of content. Position already entered through the scores.

Query compression and YaRN

V3 down-projects queries through a 1536-d latent before splitting into \(q^{C}\) and \(q^{R}\). That saves training activation memory. It does not change what is cached. Long context on the RoPE slice uses YaRN; the content latent does not need it.

512 + 64 = 576

latent plus RoPE key, times 61 layers, times two bytes for bf16.

Scale factor against MHA on this stack: \(2 \cdot 128 \cdot 128 / 576 = 4N/9 \approx 56.9\times\).

Same 32-user, 8,192-token chatbot as Deep Cuts 06 and chapter 02:

What you cache32 × 8,192
V3-shaped MHA976 GiB
MLA, latent only (ch 02)15.25 GiB
MLA + 64-d RoPE key17.16 GiB
The RoPE key adds about 2 GiB to the chatbot and keeps position. Llama 3 8B GQA on the same load is 32 GiB.

Per decode step you append one 512-vector and one already-rotated 64-vector. You do not re-rotate the cache.

512+64 · 70,272 · hold

MANIM 512 + 64 cached numbers → 70,272 bytes/token.

Shared \(k_R\), per-head \(q_R\)

Notice w_kr maps to d_rope, not to n_heads * d_rope. The shared key is expanded across heads only when scoring. Rotate before the cache write. The check script confirms a prefill-then-decode run matches a full pass, and that the RoPE cache is (batch, tokens, d_rope).

mla_rope.pycached decode = full pass; k_R is shared across heads
"""MLA with decoupled RoPE.

RoPE is a position-dependent rotation. If you rotate the reconstructed
key, that rotation sits between W_Q and W_UK and you can no longer fold
them together. DeepSeek's fix: keep a RoPE-free content path (absorbable,
cached as c_KV) and a small shared RoPE key (cached as k_R).

Per token we store d_c + d_rope numbers. For DeepSeek-V3 that is 512 + 64.
"""
import torch
import torch.nn as nn


def rotate_pairs(x, cos, sin):
    """Rotate consecutive pairs (x0, x1), (x2, x3), ... by the given angles."""
    x_even, x_odd = x[..., 0::2], x[..., 1::2]
    rot_even = x_even * cos - x_odd * sin
    rot_odd = x_even * sin + x_odd * cos
    return torch.stack((rot_even, rot_odd), dim=-1).flatten(-2)


def rope_angles(seq_len, dim, base=10_000.0, device=None):
    """cos/sin for positions 0..seq_len-1, one pair of angles per two dims."""
    # Even-odd pairing uses dim/2 frequencies.
    freqs = 1.0 / (base ** (torch.arange(0, dim, 2, device=device).float() / dim))
    pos = torch.arange(seq_len, device=device).float()
    angles = torch.outer(pos, freqs)                  # (T, dim/2)
    return torch.cos(angles), torch.sin(angles)


class DualCache:
    """Content latent and the shared RoPE key, for one layer."""

    def __init__(self):
        self.c = None   # (batch, tokens, d_c)
        self.k_r = None  # (batch, tokens, d_rope), already rotated

    def append(self, c_new, k_r_new):
        if self.c is None:
            self.c, self.k_r = c_new, k_r_new
        else:
            self.c = torch.cat([self.c, c_new], dim=1)
            self.k_r = torch.cat([self.k_r, k_r_new], dim=1)
        return self.c, self.k_r


class DecoupledMLA(nn.Module):
    """Content path through a latent; position path is a shared RoPE key."""

    def __init__(self, d_model, n_heads, d_head, d_c, d_rope):
        super().__init__()
        self.n_heads = n_heads
        self.d_head = d_head
        self.d_c = d_c
        self.d_rope = d_rope
        # Content KV: compress, then up-project. RoPE never touches these.
        self.w_dkv = nn.Linear(d_model, d_c, bias=False)
        self.w_uk = nn.Linear(d_c, n_heads * d_head, bias=False)
        self.w_uv = nn.Linear(d_c, n_heads * d_head, bias=False)
        self.rms_kv = nn.RMSNorm(d_c)
        # Shared RoPE key. One vector per token, not one per head.
        self.w_kr = nn.Linear(d_model, d_rope, bias=False)
        # Queries: content (per head) plus a RoPE slice (also per head).
        self.w_q = nn.Linear(d_model, n_heads * d_head, bias=False)
        self.w_qr = nn.Linear(d_model, n_heads * d_rope, bias=False)
        self.w_o = nn.Linear(n_heads * d_head, d_model, bias=False)

    def _split_heads(self, tensor, tokens, width):
        B = tensor.shape[0]
        return tensor.view(B, tokens, self.n_heads, width).transpose(1, 2)

    def forward(self, x, cache=None, start_pos=0):
        """x: (B, T_new, d_model). start_pos is the first new token's index."""
        B, T_new, _ = x.shape
        c_new = self.rms_kv(self.w_dkv(x))
        # Rotate the shared key by this token's position, then cache it.
        cos, sin = rope_angles(start_pos + T_new, self.d_rope, device=x.device)
        k_r_raw = self.w_kr(x)                        # (B, T_new, d_rope)
        k_r_new = rotate_pairs(k_r_raw, cos[start_pos:], sin[start_pos:])

        if cache is not None:
            c, k_r = cache.append(c_new, k_r_new)
        else:
            c, k_r = c_new, k_r_new
        T_all = c.shape[1]

        q_c = self._split_heads(self.w_q(x), T_new, self.d_head)
        k_c = self._split_heads(self.w_uk(c), T_all, self.d_head)
        v = self._split_heads(self.w_uv(c), T_all, self.d_head)

        q_r = self._split_heads(self.w_qr(x), T_new, self.d_rope)
        q_r = rotate_pairs(q_r, cos[start_pos:], sin[start_pos:])
        # Broadcast the shared RoPE key across heads. It is stored once.
        k_r_heads = k_r.unsqueeze(1).expand(-1, self.n_heads, -1, -1)

        # Concat per head: score = content_dot + rope_dot. No extra cross terms.
        q = torch.cat([q_c, q_r], dim=-1)
        k = torch.cat([k_c, k_r_heads], dim=-1)
        scale = (self.d_head + self.d_rope) ** 0.5
        scores = q @ k.transpose(-2, -1) / scale
        pos_q = torch.arange(T_all - T_new, T_all, device=x.device).unsqueeze(1)
        pos_k = torch.arange(T_all, device=x.device).unsqueeze(0)
        scores = scores.masked_fill(pos_k > pos_q, float("-inf"))
        ctx = torch.softmax(scores, dim=-1) @ v
        ctx = ctx.transpose(1, 2).contiguous().view(B, T_new, -1)
        return self.w_o(ctx)
  • rotate_pairs is even/odd RoPE. Apply it to the new slice only, using start_pos.
  • Values come only from \(c_{KV}\). Concatenation is per head: \(d_h + d_{\text{rope}}\), then one softmax per head.
  • Scale is \(\sqrt{d_h + d_{\text{rope}}}\), not \(\sqrt{d_h}\).

Mistakes learners make

Re-rotating cached \(k_R\) at the current position. Each key is rotated by its position, then stored. Re-applying the newest angle would smash relative position.

One attention over a concatenated mega-head. Concatenate per head, then run \(H\) softmaxes. A single 16-d softmax is not multi-head MLA.

“The two subspaces must be orthogonal.” Concatenation already splits the dot product. You do not need \(q^{C}\!\cdot k^{R} = 0\).

A per-head RoPE key in the cache. That would be \(128 \times 64\) extra numbers and would undo the win. V3 stores one 64-vector.

What to remember

  • RoPE between \(W_Q\) and \(W_{UK}\) kills absorption. Keep content RoPE-free.
  • Cache \(c_{KV}\) (512) and a shared rotated \(k_R\) (64). Queries carry a per-head RoPE slice. Values do not.
  • Score = content dot + RoPE dot. That is just concatenation.
  • 576 × 61 × 2 = 70,272 bytes per token. The 32 × 8,192 chatbot is 17.16 GiB vs 976 GiB of MHA.

Check yourself

1. Why can't you RoPE the reconstructed key and still absorb?

The rotation depends on the cached token's position, so there is no single matrix \(W_Q W_{UK}^{\top}\) that works for every key.

2. How many RoPE-key numbers are cached per token per layer?

64. One shared vector, not 128 heads × 64.

3. \(q^{C}=[1,2]\), \(q^{R}=[3,4]\), \(k^{C}=[0.5,0.5]\), \(k^{R}=[1,0]\). What is the score?

4.5. Sum of 1.5 and 3, which matches the concatenated dot product.

4. What is 70,272?

\((512+64) \times 61 \times 2\) bytes: the bf16 cache for one token across V3.

Go to the source

Keep going