THE EVOLVING TRANSFORMER · STOP 07

Qwen3-Next

Three linear-attention layers, then one gated softmax layer. Most of the stack keeps a constant-size state. A quarter of it keeps a KV cache.

~14 min readHybrid attentionNot weight-tied

What you'll learn

  • Gated DeltaNet: a recurrent D×D state instead of an S×S score matrix
  • Gated attention: softmax scores, sigmoid output gate, QK-RMSNorm
  • Partial RoPE on 25% of each head
  • The 80B-A3B published config: 2,048 hidden, 16 / 2 heads, 512 experts top-10

Before you start

Linear layers, then one full layer

DeepSeek compressed the cache. Qwen3-Next mostly refuses to grow one. Layers 0, 1, 2 are Gated DeltaNet: each step updates a D×D state with a gated delta rule and writes a new hidden vector. Layer 3 is Gated Attention: ordinary softmax GQA, Q and K passed through RMSNorm, RoPE on the first 25% of the head, and a sigmoid gate on the output (from a projection of x, not from the scores). Then the pattern repeats. Every layer still has a shared-expert MoE. The published 80B-A3B checkpoint is not weight-tied.

Picture a clerk who keeps one running chalkboard (the state) for three pages, then once in a while rereads the whole folder (full attention). The chalkboard size does not grow with the book. The folder does.

The teaching DeltaNet is a single-head Python loop with no short conv and no extra output gate. The real model is multi-head, with a length-4 causal conv. The page and the file say so in the same words.

Qwen3-Next-80B-A3B, then the toy stack

Knob80B-A3B
hidden / layers2,048 / 48
Q / KV / head dim16 / 2 / 256
RoPE25% of 256 = 64 dims, θ = 10,000,000
MoE512 routed, top-10, + 1 shared
vocab151,936
hybridfull_attention_interval = 4
weight tyingno

Teaching: hidden 32, 4 layers, 4 Q / 2 KV, 4 experts top-2, length 8. Layer 3 is full attention; layers 0–2 are DeltaNet. Partial RoPE rotates 2 of the 8 head dims.

linear layers · one full attn · RoPE + gate

MANIM Hybrid stack: cheap linear layers + one full softmax layer.
TensorShape
tokens(2, 8)
DeltaNet state(2, 32, 32)
full-attn q(2, 4, 8, 8)
logits(2, 8, 50)

Names on the page are names in the file

GatedDeltaNet, GatedAttention, PartialRoPE, MoE, Block, Qwen3Next. Softmax on the scores. Sigmoid on the output gate. Partial RoPE with partial_rotary_factor = 0.25.

qwen3.pyhybrid forward + partial RoPE + softmax gate
"""Qwen3-Next teaching model: linear attention + a hybrid stack.

Published Qwen3-Next-80B-A3B: hidden 2,048, 48 layers, 16 Q / 2 KV heads,
head dim 256, vocab 151,936, 512 routed experts, 10 per token + 1 shared,
partial RoPE 0.25, full_attention_interval 4 (3 linear : 1 gated softmax).
NOT weight-tied.

This file is written to match the page. Simplifications are named:
GatedDeltaNet is a single-head D×D Python loop (no short conv).
GatedAttention is softmax + a sigmoid output gate, not sigmoid scores.
Partial RoPE rotates 25% of each head.
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class Qwen3Config:
    dmodel: int = 32
    N: int = 4
    H: int = 4
    n_kv_heads: int = 2
    vocabsize: int = 50
    max_seq_len: int = 16
    dff: int = 64
    norm_eps: float = 1e-6
    num_experts: int = 4
    top_k: int = 2
    full_attn_every: int = 4
    partial_rotary_factor: float = 0.25
    rope_theta: float = 10_000_000.0

    @property
    def dk(self):
        return self.dmodel // self.H


class RMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.weight


class PartialRoPE(nn.Module):
    """Rotate only the first `factor` of the head. The rest stays untouched."""

    def __init__(self, head_dim, max_seq_len, theta, factor):
        super().__init__()
        self.rot_dim = int(head_dim * factor)
        # rot_dim must be even
        self.rot_dim = self.rot_dim - (self.rot_dim % 2)
        freqs = 1.0 / (theta ** (torch.arange(0, self.rot_dim, 2).float() / self.rot_dim))
        angles = torch.outer(torch.arange(max_seq_len).float(), freqs)
        self.register_buffer("cos_cached", torch.cos(angles))
        self.register_buffer("sin_cached", torch.sin(angles))

    def forward(self, x):
        B, H, S, D = x.shape
        rot, pass_through = x[..., : self.rot_dim], x[..., self.rot_dim :]
        even, odd = rot[..., 0::2], rot[..., 1::2]
        cos = self.cos_cached[:S].view(1, 1, S, -1)
        sin = self.sin_cached[:S].view(1, 1, S, -1)
        rotated = torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1)
        return torch.cat([rotated.view(B, H, S, self.rot_dim), pass_through], dim=-1)


class GatedDeltaNet(nn.Module):
    """Linear attention via a gated delta rule.

    Teaching simplifications (said on the page): one D×D state, no short
    conv, no output gate, Python loop over S. Real Qwen3-Next is multi-head
    with a per-head state and a length-4 causal conv.
    """

    def __init__(self, config):
        super().__init__()
        D = config.dmodel
        self.wq = nn.Linear(D, D, bias=False)
        self.wk = nn.Linear(D, D, bias=False)
        self.wv = nn.Linear(D, D, bias=False)
        self.wb = nn.Linear(D, 1, bias=True)
        self.wa = nn.Linear(D, 1, bias=True)
        self.wo = nn.Linear(D, D, bias=False)

    def forward(self, x):
        B, S, D = x.shape
        Q, K, V = self.wq(x), F.normalize(self.wk(x), dim=-1), self.wv(x)
        beta, alpha = torch.sigmoid(self.wb(x)), torch.sigmoid(self.wa(x))
        state = torch.zeros(B, D, D, device=x.device, dtype=x.dtype)
        outs = []
        for t in range(S):
            kt, vt, qt = K[:, t], V[:, t], Q[:, t]
            bt, at = beta[:, t], alpha[:, t]
            # S ← α S (I − β k kᵀ) + β v kᵀ   (erase along k, then write)
            I_minus = torch.eye(D, device=x.device) - bt.unsqueeze(-1) * kt.unsqueeze(-1) * kt.unsqueeze(-2)
            state = at.unsqueeze(-1) * state @ I_minus + bt.unsqueeze(-1) * vt.unsqueeze(-1) * kt.unsqueeze(-2)
            outs.append((state @ qt.unsqueeze(-1)).squeeze(-1))
        return self.wo(torch.stack(outs, dim=1))


class GatedAttention(nn.Module):
    """Softmax GQA + RMSNorm on Q/K + a sigmoid gate on the attention output."""

    def __init__(self, config):
        super().__init__()
        self.H = config.H
        self.n_kv = config.n_kv_heads
        self.n_rep = config.H // config.n_kv_heads
        self.dk = config.dk
        self.wq = nn.Linear(config.dmodel, config.H * config.dk, bias=False)
        self.wk = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
        self.wv = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
        self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)
        self.q_norm = RMSNorm(config.dk, config.norm_eps)
        self.k_norm = RMSNorm(config.dk, config.norm_eps)
        # output gate from x, not from the scores
        self.w_gate = nn.Linear(config.dmodel, config.dmodel, bias=False)

    def forward(self, x, rope, mask):
        B, S, _ = x.shape
        q = self.q_norm(self.wq(x).view(B, S, self.H, self.dk)).transpose(1, 2)
        k = self.k_norm(self.wk(x).view(B, S, self.n_kv, self.dk)).transpose(1, 2)
        v = self.wv(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
        q, k = rope(q), rope(k)
        if self.n_rep > 1:
            k = k.unsqueeze(2).expand(B, self.n_kv, self.n_rep, S, self.dk).reshape(B, self.H, S, self.dk)
            v = v.unsqueeze(2).expand(B, self.n_kv, self.n_rep, S, self.dk).reshape(B, self.H, S, self.dk)
        scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
        scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
        out = (scores.softmax(dim=-1) @ v).transpose(1, 2).contiguous().view(B, S, self.H * self.dk)
        return self.wo(out) * torch.sigmoid(self.w_gate(x))


class SwiGLU(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))


class MoE(nn.Module):
    """Shared expert (always on) + top-k routed experts."""

    def __init__(self, config):
        super().__init__()
        self.shared = SwiGLU(config)
        self.experts = nn.ModuleList([SwiGLU(config) for _ in range(config.num_experts)])
        self.router = nn.Linear(config.dmodel, config.num_experts, bias=False)
        self.top_k = config.top_k

    def forward(self, x):
        probs = F.softmax(self.router(x), dim=-1)
        weight, indices = torch.topk(probs, self.top_k, dim=-1)
        weight = weight / weight.sum(dim=-1, keepdim=True)
        routed = torch.zeros_like(x)
        for k in range(self.top_k):
            for i, expert in enumerate(self.experts):
                chosen = indices[:, :, k] == i
                if chosen.any():
                    routed[chosen] += weight[:, :, k][chosen].unsqueeze(-1) * expert(x[chosen])
        return self.shared(x) + routed


class Block(nn.Module):
    def __init__(self, config, layer_idx):
        super().__init__()
        # layers 3, 7, … are full attention; the rest are linear
        self.use_full_attn = (layer_idx % config.full_attn_every) == (config.full_attn_every - 1)
        self.attn = GatedAttention(config) if self.use_full_attn else GatedDeltaNet(config)
        self.moe = MoE(config)
        self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
        self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)

    def forward(self, x, rope, mask):
        h = self.rms_1(x)
        x = x + (self.attn(h, rope, mask) if self.use_full_attn else self.attn(h))
        return x + self.moe(self.rms_2(x))


class Qwen3Next(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.wte = nn.Embedding(config.vocabsize, config.dmodel)
        self.rope = PartialRoPE(config.dk, config.max_seq_len, config.rope_theta, config.partial_rotary_factor)
        self.blocks = nn.ModuleList([Block(config, i) for i in range(config.N)])
        self.norm = RMSNorm(config.dmodel, config.norm_eps)
        self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
        self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)))

    def forward(self, idx):
        x = self.wte(idx)
        for block in self.blocks:
            x = block(x, self.rope, self.mask)
        return self.lm_head(self.norm(x))

Test note. Forward (2, 8), assert logits, check layers 0–2 are GatedDeltaNet and layer 3 is GatedAttention, check rot_dim == 2, and confirm the head is not tied.

Mistakes this stop used to make

Replacing softmax with sigmoid on the scores. The gate is on the attention output, from a linear of x. Scores stay softmax.

Rotating the whole head. Only 25% of the dims take RoPE (64 of 256 published; 2 of 8 in the toy).

Copying the old mismatched class names. There is no MoEWithSharedExpert or build_qwen3_next in this file.

Publishing d_model 5,120 / 40 heads / 128 experts. Those are not the 80B-A3B numbers.

What to remember

  • 3 linear layers + 1 gated softmax layer. KV cache only on the softmax layers.
  • Gated attention = softmax + QK-RMSNorm + sigmoid output gate + partial RoPE.
  • 80B-A3B: hidden 2,048, 16 / 2 / 256, 512 experts top-10 + 1 shared, vocab 151,936, not tied.

Check yourself

1. In a 4-layer toy, which layer is full attention?

Layer 3 — the last of each group of 4.

2. What does the sigmoid gate multiply?

The attention output, not the scores.

3. How many experts fire per token in 80B-A3B?

10 routed, plus 1 shared.

Go to the source

Keep going