THE EVOLVING TRANSFORMER · STOP 06
GQA is gone. Keys and values compress into a small latent. Routing is top-8 of 256 plus one shared expert. One extra next-token head. Released December 2024.
Mixtral shared KV heads. DeepSeek-V3 does not use GQA. It compresses each token's keys and values into a latent c_kv of 512 numbers, plus a 64-dim RoPE key that is shared across heads and broadcast. That is 576 numbers cached per token per layer — 70,272 bytes for the whole 61-layer model in bf16. The first layer is a dense SwiGLU. The rest are DeepSeekMoE: 256 routed experts, top-8, plus 1 shared expert that always runs. Affinities are sigmoid, not softmax. A learned bias steers load without entering the gate weights. One MTP module sits on the final hidden state. The MTP loss weight λ is 0.3, then 0.1.
Instead of eight shared card boxes (GQA), the clerk writes a short index card per page and one position sticker. At read time the card expands back into a full set of keys and values. The sticker is the same for every clerk.
MLA, decoupled RoPE, DeepSeekMoE and MTP each have a Decoding DeepSeek chapter. This stop is the one file those chapters build. DeepSeek in one page →
| Fact | DeepSeek-V3 |
|---|---|
| released | December 2024 |
| parameters | 671B total, 37B active |
| layers | 61 (first is dense) |
| MoE | 256 routed + 1 shared, top-8 |
| MLA cache | 512 + 64 = 576 numbers / token / layer |
| bytes / token (bf16) | 70,272 |
| MTP | one module, λ = 0.3 then 0.1 |
| GQA | no — MLA instead |
Teaching batch: hidden 32, 4 heads, latent 8, RoPE key 4, length 8, 8 routed experts top-2, first layer dense.
MLA latent · MoE top-8 · MTP
| Tensor | Shape |
|---|---|
| tokens | (2, 8) |
c_kv | (2, 8, 8) |
| shared RoPE key before broadcast | (2, 1, 8, 4) |
| main logits / MTP logits | (2, 8, 50) |
"""DeepSeek-V3 teaching model: MLA + DeepSeekMoE + one MTP head.
Published facts (December 2024) — these win over any older page:
671B total / 37B active, 61 layers (first is dense),
256 routed experts + 1 shared, top-8 (not top-6),
MLA caches 512 latent + 64 RoPE key = 576 numbers per token per layer,
one MTP module, loss weight λ = 0.3 then 0.1.
Does NOT use GQA.
Teaching default is tiny. The RoPE key is shared across heads
(dmodel → rope_head_dim), then broadcast — that is what keeps the
cache small. Routing is sigmoid + a learned bias (aux-loss-free).
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class DeepseekConfig:
dmodel: int = 32
N: int = 2
H: int = 4
vocabsize: int = 50
max_seq_len: int = 16
norm_eps: float = 1e-6
# MoE — scaled down from 256 routed + 1 shared, top-8
num_experts: int = 8
top_k: int = 2
n_dense_layers: int = 1
# MLA — scaled from 512 latent + 64 RoPE
kv_lora_rank: int = 8
q_lora_rank: int = 16
rope_head_dim: int = 4
nope_head_dim: int = 4
mtp_lambda: float = 0.3
@property
def dk(self):
return self.rope_head_dim + self.nope_head_dim
@property
def dff(self):
return 4 * self.dmodel
class RMSNorm(nn.Module):
def __init__(self, dim, eps=1e-6):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def forward(self, x):
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight
class SwiGLU(nn.Module):
def __init__(self, config):
super().__init__()
self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)
def forward(self, x):
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
class DeepseekRouter(nn.Module):
"""Sigmoid affinities + a bias used only for top-k (not for weights)."""
def __init__(self, config):
super().__init__()
self.top_k = config.top_k
self.router = nn.Linear(config.dmodel, config.num_experts, bias=False)
self.bias = nn.Parameter(torch.zeros(config.num_experts))
def forward(self, x):
affinity = torch.sigmoid(self.router(x)) # (B, S, E)
# bias steers load; it is NOT added into the gate weights
_, indices = torch.topk(affinity + self.bias, self.top_k, dim=-1)
chosen = torch.gather(affinity, -1, indices)
weight = chosen / chosen.sum(dim=-1, keepdim=True).clamp_min(1e-9)
return weight, indices
class DeepseekMoE(nn.Module):
def __init__(self, config):
super().__init__()
self.shared = SwiGLU(config) # always on
self.experts = nn.ModuleList([SwiGLU(config) for _ in range(config.num_experts)])
self.router = DeepseekRouter(config)
self.top_k = config.top_k
def forward(self, x):
weights, indices = self.router(x)
routed = torch.zeros_like(x)
for k in range(self.top_k):
for i, expert in enumerate(self.experts):
chosen = indices[:, :, k] == i
if chosen.any():
routed[chosen] += weights[:, :, k][chosen].unsqueeze(-1) * expert(x[chosen])
return self.shared(x) + routed
class RotaryPositionalEmbedding(nn.Module):
def __init__(self, head_dim, max_seq_len, theta=10_000.0):
super().__init__()
freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
angles = torch.outer(torch.arange(max_seq_len).float(), freqs)
self.register_buffer("cos_cached", torch.cos(angles))
self.register_buffer("sin_cached", torch.sin(angles))
def forward(self, x):
B, H, S, _ = x.shape
even, odd = x[..., 0::2], x[..., 1::2]
cos = self.cos_cached[:S].view(1, 1, S, -1)
sin = self.sin_cached[:S].view(1, 1, S, -1)
return torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1).view(B, H, S, -1)
class MultiHeadLatentAttention(nn.Module):
"""Compress KV to c_kv; a single shared RoPE key is broadcast to all heads."""
def __init__(self, config):
super().__init__()
self.H = config.H
self.dk = config.dk
self.nope = config.nope_head_dim
self.rope_d = config.rope_head_dim
self.kv_rank = config.kv_lora_rank
self.w_down_kv = nn.Linear(config.dmodel, config.kv_lora_rank, bias=False)
self.w_up_k = nn.Linear(config.kv_lora_rank, config.H * config.nope_head_dim, bias=False)
self.w_up_v = nn.Linear(config.kv_lora_rank, config.H * config.nope_head_dim, bias=False)
# shared RoPE key: one vector per token, not per head
self.w_kr = nn.Linear(config.dmodel, config.rope_head_dim, bias=False)
self.w_down_q = nn.Linear(config.dmodel, config.q_lora_rank, bias=False)
self.w_up_q = nn.Linear(config.q_lora_rank, config.H * config.nope_head_dim, bias=False)
self.w_qr = nn.Linear(config.q_lora_rank, config.H * config.rope_head_dim, bias=False)
self.wo = nn.Linear(config.H * config.nope_head_dim, config.dmodel, bias=False)
def forward(self, x, rope, mask):
B, S, _ = x.shape
c_kv = self.w_down_kv(x) # (B, S, kv_lora_rank) — this is what you cache
k_nope = self.w_up_k(c_kv).view(B, S, self.H, self.nope).transpose(1, 2)
v = self.w_up_v(c_kv).view(B, S, self.H, self.nope).transpose(1, 2)
# (B, S, rope_d) → (B, 1, S, rope_d) → broadcast over heads
k_rope = rope(self.w_kr(x).view(B, S, 1, self.rope_d).transpose(1, 2))
k_rope = k_rope.expand(-1, self.H, -1, -1)
k = torch.cat([k_nope, k_rope], dim=-1)
c_q = self.w_down_q(x)
q_nope = self.w_up_q(c_q).view(B, S, self.H, self.nope).transpose(1, 2)
q_rope = rope(self.w_qr(c_q).view(B, S, self.H, self.rope_d).transpose(1, 2))
q = torch.cat([q_nope, q_rope], dim=-1)
scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
out = scores.softmax(dim=-1) @ v
return self.wo(out.transpose(1, 2).contiguous().view(B, S, self.H * self.nope)), c_kv
class Block(nn.Module):
def __init__(self, config, layer_idx):
super().__init__()
self.mla = MultiHeadLatentAttention(config)
self.ff = SwiGLU(config) if layer_idx < config.n_dense_layers else DeepseekMoE(config)
self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)
def forward(self, x, rope, mask):
attn_out, c_kv = self.mla(self.rms_1(x), rope, mask)
x = x + attn_out
return x + self.ff(self.rms_2(x)), c_kv
class MTP(nn.Module):
"""One extra next-token head on the final hidden state (DeepSeek uses one)."""
def __init__(self, config):
super().__init__()
self.norm = RMSNorm(config.dmodel, config.norm_eps)
self.head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
def forward(self, hidden):
return self.head(self.norm(hidden))
class DeepseekV3(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.wte = nn.Embedding(config.vocabsize, config.dmodel)
self.rope = RotaryPositionalEmbedding(config.rope_head_dim, config.max_seq_len)
self.blocks = nn.ModuleList([Block(config, i) for i in range(config.N)])
self.norm = RMSNorm(config.dmodel, config.norm_eps)
self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
self.mtp = MTP(config)
self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)))
def forward(self, idx):
x = self.wte(idx)
latents = []
for block in self.blocks:
x, c_kv = block(x, self.rope, self.mask)
latents.append(c_kv)
hidden = self.norm(x)
return self.lm_head(hidden), self.mtp(hidden), latents
Test note. Forward a tiny batch, assert c_kv and both logit tensors, check w_kr is shared (not per-head), layer 0 is dense, layer 1 is MoE, and there is no GQA module.
Top-6 of 256. That was V2-shaped folklore. V3 is top-8 of 256 routed + 1 shared.
A RoPE key per head. One shared d_model → 64 projection, then broadcast. Per-head RoPE keys swell the cache.
Calling it GQA. DeepSeek-V3 uses MLA. GQA is the previous stop.
Leftover model.py / Bumblebee diffs. The previous stop is Mixtral in this series, not another codebase.
576 — 512 latent plus a 64-dim RoPE key.
Top-8 of 256 routed, plus 1 shared.
No. It uses MLA.