THE EVOLVING TRANSFORMER · STOP 07
Three linear-attention layers, then one gated softmax layer. Most of the stack keeps a constant-size state. A quarter of it keeps a KV cache.
DeepSeek compressed the cache. Qwen3-Next mostly refuses to grow one. Layers 0, 1, 2 are Gated DeltaNet: each step updates a D×D state with a gated delta rule and writes a new hidden vector. Layer 3 is Gated Attention: ordinary softmax GQA, Q and K passed through RMSNorm, RoPE on the first 25% of the head, and a sigmoid gate on the output (from a projection of x, not from the scores). Then the pattern repeats. Every layer still has a shared-expert MoE. The published 80B-A3B checkpoint is not weight-tied.
Picture a clerk who keeps one running chalkboard (the state) for three pages, then once in a while rereads the whole folder (full attention). The chalkboard size does not grow with the book. The folder does.
The teaching DeltaNet is a single-head Python loop with no short conv and no extra output gate. The real model is multi-head, with a length-4 causal conv. The page and the file say so in the same words.
| Knob | 80B-A3B |
|---|---|
| hidden / layers | 2,048 / 48 |
| Q / KV / head dim | 16 / 2 / 256 |
| RoPE | 25% of 256 = 64 dims, θ = 10,000,000 |
| MoE | 512 routed, top-10, + 1 shared |
| vocab | 151,936 |
| hybrid | full_attention_interval = 4 |
| weight tying | no |
Teaching: hidden 32, 4 layers, 4 Q / 2 KV, 4 experts top-2, length 8. Layer 3 is full attention; layers 0–2 are DeltaNet. Partial RoPE rotates 2 of the 8 head dims.
linear layers · one full attn · RoPE + gate
| Tensor | Shape |
|---|---|
| tokens | (2, 8) |
| DeltaNet state | (2, 32, 32) |
| full-attn q | (2, 4, 8, 8) |
| logits | (2, 8, 50) |
GatedDeltaNet, GatedAttention, PartialRoPE, MoE, Block, Qwen3Next. Softmax on the scores. Sigmoid on the output gate. Partial RoPE with partial_rotary_factor = 0.25.
"""Qwen3-Next teaching model: linear attention + a hybrid stack.
Published Qwen3-Next-80B-A3B: hidden 2,048, 48 layers, 16 Q / 2 KV heads,
head dim 256, vocab 151,936, 512 routed experts, 10 per token + 1 shared,
partial RoPE 0.25, full_attention_interval 4 (3 linear : 1 gated softmax).
NOT weight-tied.
This file is written to match the page. Simplifications are named:
GatedDeltaNet is a single-head D×D Python loop (no short conv).
GatedAttention is softmax + a sigmoid output gate, not sigmoid scores.
Partial RoPE rotates 25% of each head.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class Qwen3Config:
dmodel: int = 32
N: int = 4
H: int = 4
n_kv_heads: int = 2
vocabsize: int = 50
max_seq_len: int = 16
dff: int = 64
norm_eps: float = 1e-6
num_experts: int = 4
top_k: int = 2
full_attn_every: int = 4
partial_rotary_factor: float = 0.25
rope_theta: float = 10_000_000.0
@property
def dk(self):
return self.dmodel // self.H
class RMSNorm(nn.Module):
def __init__(self, dim, eps=1e-6):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def forward(self, x):
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight
class PartialRoPE(nn.Module):
"""Rotate only the first `factor` of the head. The rest stays untouched."""
def __init__(self, head_dim, max_seq_len, theta, factor):
super().__init__()
self.rot_dim = int(head_dim * factor)
# rot_dim must be even
self.rot_dim = self.rot_dim - (self.rot_dim % 2)
freqs = 1.0 / (theta ** (torch.arange(0, self.rot_dim, 2).float() / self.rot_dim))
angles = torch.outer(torch.arange(max_seq_len).float(), freqs)
self.register_buffer("cos_cached", torch.cos(angles))
self.register_buffer("sin_cached", torch.sin(angles))
def forward(self, x):
B, H, S, D = x.shape
rot, pass_through = x[..., : self.rot_dim], x[..., self.rot_dim :]
even, odd = rot[..., 0::2], rot[..., 1::2]
cos = self.cos_cached[:S].view(1, 1, S, -1)
sin = self.sin_cached[:S].view(1, 1, S, -1)
rotated = torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1)
return torch.cat([rotated.view(B, H, S, self.rot_dim), pass_through], dim=-1)
class GatedDeltaNet(nn.Module):
"""Linear attention via a gated delta rule.
Teaching simplifications (said on the page): one D×D state, no short
conv, no output gate, Python loop over S. Real Qwen3-Next is multi-head
with a per-head state and a length-4 causal conv.
"""
def __init__(self, config):
super().__init__()
D = config.dmodel
self.wq = nn.Linear(D, D, bias=False)
self.wk = nn.Linear(D, D, bias=False)
self.wv = nn.Linear(D, D, bias=False)
self.wb = nn.Linear(D, 1, bias=True)
self.wa = nn.Linear(D, 1, bias=True)
self.wo = nn.Linear(D, D, bias=False)
def forward(self, x):
B, S, D = x.shape
Q, K, V = self.wq(x), F.normalize(self.wk(x), dim=-1), self.wv(x)
beta, alpha = torch.sigmoid(self.wb(x)), torch.sigmoid(self.wa(x))
state = torch.zeros(B, D, D, device=x.device, dtype=x.dtype)
outs = []
for t in range(S):
kt, vt, qt = K[:, t], V[:, t], Q[:, t]
bt, at = beta[:, t], alpha[:, t]
# S ← α S (I − β k kᵀ) + β v kᵀ (erase along k, then write)
I_minus = torch.eye(D, device=x.device) - bt.unsqueeze(-1) * kt.unsqueeze(-1) * kt.unsqueeze(-2)
state = at.unsqueeze(-1) * state @ I_minus + bt.unsqueeze(-1) * vt.unsqueeze(-1) * kt.unsqueeze(-2)
outs.append((state @ qt.unsqueeze(-1)).squeeze(-1))
return self.wo(torch.stack(outs, dim=1))
class GatedAttention(nn.Module):
"""Softmax GQA + RMSNorm on Q/K + a sigmoid gate on the attention output."""
def __init__(self, config):
super().__init__()
self.H = config.H
self.n_kv = config.n_kv_heads
self.n_rep = config.H // config.n_kv_heads
self.dk = config.dk
self.wq = nn.Linear(config.dmodel, config.H * config.dk, bias=False)
self.wk = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
self.wv = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)
self.q_norm = RMSNorm(config.dk, config.norm_eps)
self.k_norm = RMSNorm(config.dk, config.norm_eps)
# output gate from x, not from the scores
self.w_gate = nn.Linear(config.dmodel, config.dmodel, bias=False)
def forward(self, x, rope, mask):
B, S, _ = x.shape
q = self.q_norm(self.wq(x).view(B, S, self.H, self.dk)).transpose(1, 2)
k = self.k_norm(self.wk(x).view(B, S, self.n_kv, self.dk)).transpose(1, 2)
v = self.wv(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
q, k = rope(q), rope(k)
if self.n_rep > 1:
k = k.unsqueeze(2).expand(B, self.n_kv, self.n_rep, S, self.dk).reshape(B, self.H, S, self.dk)
v = v.unsqueeze(2).expand(B, self.n_kv, self.n_rep, S, self.dk).reshape(B, self.H, S, self.dk)
scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
out = (scores.softmax(dim=-1) @ v).transpose(1, 2).contiguous().view(B, S, self.H * self.dk)
return self.wo(out) * torch.sigmoid(self.w_gate(x))
class SwiGLU(nn.Module):
def __init__(self, config):
super().__init__()
self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)
def forward(self, x):
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
class MoE(nn.Module):
"""Shared expert (always on) + top-k routed experts."""
def __init__(self, config):
super().__init__()
self.shared = SwiGLU(config)
self.experts = nn.ModuleList([SwiGLU(config) for _ in range(config.num_experts)])
self.router = nn.Linear(config.dmodel, config.num_experts, bias=False)
self.top_k = config.top_k
def forward(self, x):
probs = F.softmax(self.router(x), dim=-1)
weight, indices = torch.topk(probs, self.top_k, dim=-1)
weight = weight / weight.sum(dim=-1, keepdim=True)
routed = torch.zeros_like(x)
for k in range(self.top_k):
for i, expert in enumerate(self.experts):
chosen = indices[:, :, k] == i
if chosen.any():
routed[chosen] += weight[:, :, k][chosen].unsqueeze(-1) * expert(x[chosen])
return self.shared(x) + routed
class Block(nn.Module):
def __init__(self, config, layer_idx):
super().__init__()
# layers 3, 7, … are full attention; the rest are linear
self.use_full_attn = (layer_idx % config.full_attn_every) == (config.full_attn_every - 1)
self.attn = GatedAttention(config) if self.use_full_attn else GatedDeltaNet(config)
self.moe = MoE(config)
self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)
def forward(self, x, rope, mask):
h = self.rms_1(x)
x = x + (self.attn(h, rope, mask) if self.use_full_attn else self.attn(h))
return x + self.moe(self.rms_2(x))
class Qwen3Next(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.wte = nn.Embedding(config.vocabsize, config.dmodel)
self.rope = PartialRoPE(config.dk, config.max_seq_len, config.rope_theta, config.partial_rotary_factor)
self.blocks = nn.ModuleList([Block(config, i) for i in range(config.N)])
self.norm = RMSNorm(config.dmodel, config.norm_eps)
self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)))
def forward(self, idx):
x = self.wte(idx)
for block in self.blocks:
x = block(x, self.rope, self.mask)
return self.lm_head(self.norm(x))
Test note. Forward (2, 8), assert logits, check layers 0–2 are GatedDeltaNet and layer 3 is GatedAttention, check rot_dim == 2, and confirm the head is not tied.
Replacing softmax with sigmoid on the scores. The gate is on the attention output, from a linear of x. Scores stay softmax.
Rotating the whole head. Only 25% of the dims take RoPE (64 of 256 published; 2 of 8 in the toy).
Copying the old mismatched class names. There is no MoEWithSharedExpert or build_qwen3_next in this file.
Publishing d_model 5,120 / 40 heads / 128 experts. Those are not the 80B-A3B numbers.
Layer 3 — the last of each group of 4.
The attention output, not the scores.
10 routed, plus 1 shared.