THE EVOLVING TRANSFORMER · STOP 06

DeepSeek-V3

GQA is gone. Keys and values compress into a small latent. Routing is top-8 of 256 plus one shared expert. One extra next-token head. Released December 2024.

~14 min readMLA, not GQADecember 2024

What you'll learn

  • MLA: cache 512 latent + 64 RoPE key = 576 numbers per token per layer
  • The RoPE key is shared across heads, then broadcast
  • 256 routed experts + 1 shared, top-8, sigmoid + bias
  • One MTP module, λ = 0.3 then 0.1

Before you start

Compress the cache, then split the experts

Mixtral shared KV heads. DeepSeek-V3 does not use GQA. It compresses each token's keys and values into a latent c_kv of 512 numbers, plus a 64-dim RoPE key that is shared across heads and broadcast. That is 576 numbers cached per token per layer — 70,272 bytes for the whole 61-layer model in bf16. The first layer is a dense SwiGLU. The rest are DeepSeekMoE: 256 routed experts, top-8, plus 1 shared expert that always runs. Affinities are sigmoid, not softmax. A learned bias steers load without entering the gate weights. One MTP module sits on the final hidden state. The MTP loss weight λ is 0.3, then 0.1.

Instead of eight shared card boxes (GQA), the clerk writes a short index card per page and one position sticker. At read time the card expands back into a full set of keys and values. The sticker is the same for every clerk.

MLA, decoupled RoPE, DeepSeekMoE and MTP each have a Decoding DeepSeek chapter. This stop is the one file those chapters build. DeepSeek in one page →

The December 2024 facts box

FactDeepSeek-V3
releasedDecember 2024
parameters671B total, 37B active
layers61 (first is dense)
MoE256 routed + 1 shared, top-8
MLA cache512 + 64 = 576 numbers / token / layer
bytes / token (bf16)70,272
MTPone module, λ = 0.3 then 0.1
GQAno — MLA instead

Teaching batch: hidden 32, 4 heads, latent 8, RoPE key 4, length 8, 8 routed experts top-2, first layer dense.

MLA latent · MoE top-8 · MTP

MANIM Compress KV to a latent; 256+1 MoE top-8; MTP on the side.
TensorShape
tokens(2, 8)
c_kv(2, 8, 8)
shared RoPE key before broadcast(2, 1, 8, 4)
main logits / MTP logits(2, 8, 50)

Shared RoPE key, sigmoid router, one MTP

deepseek_v3.pyMLA latent + MTP + dense-then-MoE
"""DeepSeek-V3 teaching model: MLA + DeepSeekMoE + one MTP head.

Published facts (December 2024) — these win over any older page:
  671B total / 37B active, 61 layers (first is dense),
  256 routed experts + 1 shared, top-8 (not top-6),
  MLA caches 512 latent + 64 RoPE key = 576 numbers per token per layer,
  one MTP module, loss weight λ = 0.3 then 0.1.
  Does NOT use GQA.

Teaching default is tiny. The RoPE key is shared across heads
(dmodel → rope_head_dim), then broadcast — that is what keeps the
cache small. Routing is sigmoid + a learned bias (aux-loss-free).
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class DeepseekConfig:
    dmodel: int = 32
    N: int = 2
    H: int = 4
    vocabsize: int = 50
    max_seq_len: int = 16
    norm_eps: float = 1e-6
    # MoE — scaled down from 256 routed + 1 shared, top-8
    num_experts: int = 8
    top_k: int = 2
    n_dense_layers: int = 1
    # MLA — scaled from 512 latent + 64 RoPE
    kv_lora_rank: int = 8
    q_lora_rank: int = 16
    rope_head_dim: int = 4
    nope_head_dim: int = 4
    mtp_lambda: float = 0.3

    @property
    def dk(self):
        return self.rope_head_dim + self.nope_head_dim

    @property
    def dff(self):
        return 4 * self.dmodel


class RMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight


class SwiGLU(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))


class DeepseekRouter(nn.Module):
    """Sigmoid affinities + a bias used only for top-k (not for weights)."""

    def __init__(self, config):
        super().__init__()
        self.top_k = config.top_k
        self.router = nn.Linear(config.dmodel, config.num_experts, bias=False)
        self.bias = nn.Parameter(torch.zeros(config.num_experts))

    def forward(self, x):
        affinity = torch.sigmoid(self.router(x))  # (B, S, E)
        # bias steers load; it is NOT added into the gate weights
        _, indices = torch.topk(affinity + self.bias, self.top_k, dim=-1)
        chosen = torch.gather(affinity, -1, indices)
        weight = chosen / chosen.sum(dim=-1, keepdim=True).clamp_min(1e-9)
        return weight, indices


class DeepseekMoE(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.shared = SwiGLU(config)  # always on
        self.experts = nn.ModuleList([SwiGLU(config) for _ in range(config.num_experts)])
        self.router = DeepseekRouter(config)
        self.top_k = config.top_k

    def forward(self, x):
        weights, indices = self.router(x)
        routed = torch.zeros_like(x)
        for k in range(self.top_k):
            for i, expert in enumerate(self.experts):
                chosen = indices[:, :, k] == i
                if chosen.any():
                    routed[chosen] += weights[:, :, k][chosen].unsqueeze(-1) * expert(x[chosen])
        return self.shared(x) + routed


class RotaryPositionalEmbedding(nn.Module):
    def __init__(self, head_dim, max_seq_len, theta=10_000.0):
        super().__init__()
        freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
        angles = torch.outer(torch.arange(max_seq_len).float(), freqs)
        self.register_buffer("cos_cached", torch.cos(angles))
        self.register_buffer("sin_cached", torch.sin(angles))

    def forward(self, x):
        B, H, S, _ = x.shape
        even, odd = x[..., 0::2], x[..., 1::2]
        cos = self.cos_cached[:S].view(1, 1, S, -1)
        sin = self.sin_cached[:S].view(1, 1, S, -1)
        return torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1).view(B, H, S, -1)


class MultiHeadLatentAttention(nn.Module):
    """Compress KV to c_kv; a single shared RoPE key is broadcast to all heads."""

    def __init__(self, config):
        super().__init__()
        self.H = config.H
        self.dk = config.dk
        self.nope = config.nope_head_dim
        self.rope_d = config.rope_head_dim
        self.kv_rank = config.kv_lora_rank
        self.w_down_kv = nn.Linear(config.dmodel, config.kv_lora_rank, bias=False)
        self.w_up_k = nn.Linear(config.kv_lora_rank, config.H * config.nope_head_dim, bias=False)
        self.w_up_v = nn.Linear(config.kv_lora_rank, config.H * config.nope_head_dim, bias=False)
        # shared RoPE key: one vector per token, not per head
        self.w_kr = nn.Linear(config.dmodel, config.rope_head_dim, bias=False)
        self.w_down_q = nn.Linear(config.dmodel, config.q_lora_rank, bias=False)
        self.w_up_q = nn.Linear(config.q_lora_rank, config.H * config.nope_head_dim, bias=False)
        self.w_qr = nn.Linear(config.q_lora_rank, config.H * config.rope_head_dim, bias=False)
        self.wo = nn.Linear(config.H * config.nope_head_dim, config.dmodel, bias=False)

    def forward(self, x, rope, mask):
        B, S, _ = x.shape
        c_kv = self.w_down_kv(x)  # (B, S, kv_lora_rank) — this is what you cache
        k_nope = self.w_up_k(c_kv).view(B, S, self.H, self.nope).transpose(1, 2)
        v = self.w_up_v(c_kv).view(B, S, self.H, self.nope).transpose(1, 2)
        # (B, S, rope_d) → (B, 1, S, rope_d) → broadcast over heads
        k_rope = rope(self.w_kr(x).view(B, S, 1, self.rope_d).transpose(1, 2))
        k_rope = k_rope.expand(-1, self.H, -1, -1)
        k = torch.cat([k_nope, k_rope], dim=-1)

        c_q = self.w_down_q(x)
        q_nope = self.w_up_q(c_q).view(B, S, self.H, self.nope).transpose(1, 2)
        q_rope = rope(self.w_qr(c_q).view(B, S, self.H, self.rope_d).transpose(1, 2))
        q = torch.cat([q_nope, q_rope], dim=-1)

        scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
        scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
        out = scores.softmax(dim=-1) @ v
        return self.wo(out.transpose(1, 2).contiguous().view(B, S, self.H * self.nope)), c_kv


class Block(nn.Module):
    def __init__(self, config, layer_idx):
        super().__init__()
        self.mla = MultiHeadLatentAttention(config)
        self.ff = SwiGLU(config) if layer_idx < config.n_dense_layers else DeepseekMoE(config)
        self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
        self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)

    def forward(self, x, rope, mask):
        attn_out, c_kv = self.mla(self.rms_1(x), rope, mask)
        x = x + attn_out
        return x + self.ff(self.rms_2(x)), c_kv


class MTP(nn.Module):
    """One extra next-token head on the final hidden state (DeepSeek uses one)."""

    def __init__(self, config):
        super().__init__()
        self.norm = RMSNorm(config.dmodel, config.norm_eps)
        self.head = nn.Linear(config.dmodel, config.vocabsize, bias=False)

    def forward(self, hidden):
        return self.head(self.norm(hidden))


class DeepseekV3(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.wte = nn.Embedding(config.vocabsize, config.dmodel)
        self.rope = RotaryPositionalEmbedding(config.rope_head_dim, config.max_seq_len)
        self.blocks = nn.ModuleList([Block(config, i) for i in range(config.N)])
        self.norm = RMSNorm(config.dmodel, config.norm_eps)
        self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
        self.mtp = MTP(config)
        self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)))

    def forward(self, idx):
        x = self.wte(idx)
        latents = []
        for block in self.blocks:
            x, c_kv = block(x, self.rope, self.mask)
            latents.append(c_kv)
        hidden = self.norm(x)
        return self.lm_head(hidden), self.mtp(hidden), latents

Test note. Forward a tiny batch, assert c_kv and both logit tensors, check w_kr is shared (not per-head), layer 0 is dense, layer 1 is MoE, and there is no GQA module.

Mistakes this stop used to make

Top-6 of 256. That was V2-shaped folklore. V3 is top-8 of 256 routed + 1 shared.

A RoPE key per head. One shared d_model → 64 projection, then broadcast. Per-head RoPE keys swell the cache.

Calling it GQA. DeepSeek-V3 uses MLA. GQA is the previous stop.

Leftover model.py / Bumblebee diffs. The previous stop is Mixtral in this series, not another codebase.

What to remember

  • December 2024: 671B / 37B, 61 layers, MLA not GQA.
  • Cache: 512 + 64 = 576 numbers per token per layer (70,272 bytes bf16).
  • MoE: 256 + 1 shared, top-8, sigmoid + bias. One MTP, λ = 0.3 then 0.1.

Check yourself

1. How many numbers does V3 cache per token per layer?

576 — 512 latent plus a 64-dim RoPE key.

2. Top-k of how many routed experts, plus how many shared?

Top-8 of 256 routed, plus 1 shared.

3. Does this model use GQA?

No. It uses MLA.

Go to the source

Keep going