THE TRANSFORMER DEEP CUTS · CHAPTER 07
Replace one FFN with many, and let a router pick a few per token. Most of the extra parameters stay asleep.
A harbour radio has four specialists on call: weather, traffic, repairs, stores. Each incoming phrase is handed to two of them. The desk mixes their two replies. The other two specialists sit this one out.
That is sparse MoE. The specialists are copies of the FFN. The desk is a tiny linear router. Attention is unchanged and still runs for every token.
DeepSeekMoE — fine-grained experts, a shared expert, loss-free bias — is a later design. It has its own series. This chapter is the generic router those papers start from.
For each token the router writes \(N\) logits. Keep the top \(k\). Fill the rest with \(-\infty\). Softmax what remains. The output is a weighted sum of the \(k\) chosen FFNs, and it has the same shape as a dense FFN, so the residual still fits.
Token rain, three experts, top-2, logits \([2.0,\, 0.0,\, 1.0]\):
| E0 | E1 | E2 | |
|---|---|---|---|
| logit | 2.0 | 0.0 | 1.0 |
| after top-2 softmax | 0.731 | 0 | 0.269 |
E1 is dropped. The mix is \(0.731\, E_0(x) + 0.269\, E_2(x)\).
Interactive · A three-expert desk
Move the three logits. Which two experts stay on, and at what weight?
A router that already likes expert 0 sends that expert more tokens, so it gets more gradient, so the router likes it more. Collapse. The Switch Transformer adds
\(f_i\) is the fraction of tokens that actually picked expert \(i\) (hard). \(P_i\) is the mean full-softmax probability of expert \(i\) (soft). The product is smallest when both are flat.
Eight tokens, four experts, top-2. Balanced: each expert is picked by half the tokens, \(P_i = 1/4\), and \(L_{\mathrm{aux}} = 2\) — equal to \(k\). Collapsed onto experts 0 and 1: \(L_{\mathrm{aux}} = 4\). The extra loss pushes the router back toward even traffic.
\(P_i\) is taken from the full softmax, not from the top-k gates. That is the Switch paper. Using the gates instead misses the gradient that would have gone to a dropped expert.
balanced \(L=2\) → collapse → \(L=4\) → push flat
expected load, times a slack factor. CF > 1 makes the bucket larger. Tokens over capacity are dropped (or rerouted).
Eight tokens, top-2, four experts, CF \(= 1.25\): \(C = (8 \cdot 2 / 4) \cdot 1.25 = 5\). An older write-up inverted this and made CF > 1 shrink the bucket. That is the wrong way around.
T·k/N → CF=1.25 → C=5 → overflow
A routed expert competes for tokens. A shared expert always runs, so common work ("how English works") does not have to be copied into every specialist. Mixtral 8×7B has no shared expert: 8 experts, top-2, all SwiGLU. About 46.7B total parameters, about 12.9B active. That is not \(8 \times 7.24\)B, because attention, embeddings, and norms are shared.
DeepSeek-V3, as a pointer only: 256 routed + 1 shared, top-8, 671B total, 37B active. The extra ideas (fine-grained split, loss-free bias, device-limited routing) stay in Decoding DeepSeek.
The Python walks every expert, even the ones with weight 0. A production kernel gathers the tokens of each expert instead. The numbers match. Mixtral's experts are SwiGLU; the file uses GELU so it does not depend on chapter 05.
"""A generic top-k mixture of experts. Not DeepSeekMoE.
DeepSeek's fine-grained experts, shared-expert layout, and loss-free
bias live in Decoding DeepSeek. This file is the idea those papers start from:
N copies of an FFN, a router, top-k, and a load-balancing loss.
"""
import torch
import torch.nn.functional as F
def topk_route(logits, k):
"""logits: (..., n_experts) -> gates (same shape, zeros on the unpicked).
Softmax runs on the kept experts only. A dropped expert gets weight 0
and no gradient through the softmax (its logit was never selected).
"""
values, index = torch.topk(logits, k, dim=-1)
# -inf on the unpicked slots, then softmax over experts.
masked = torch.full_like(logits, float("-inf")).scatter(-1, index, values)
return F.softmax(masked, dim=-1)
def switch_aux_loss(router_probs, gates, n_experts):
"""Switch-Transformer load-balancing loss: N * sum_i f_i P_i.
P_i is the mean *full* softmax probability of expert i (soft).
f_i is the fraction of tokens that actually picked expert i (hard).
The product is smallest when every expert is equally popular.
"""
# router_probs: (T, N) full softmax. gates: (T, N) after top-k.
p = router_probs.mean(dim=0) # (N,)
# A token "picked" an expert when that expert's gate is > 0.
f = (gates > 0).float().mean(dim=0) # (N,)
return n_experts * (f * p).sum()
class MoE(torch.nn.Module):
"""N SwiGLU-shaped experts (linear + GELU + linear) and a top-k router.
The teaching loop walks every expert. A production kernel batches the
tokens of each expert instead. The numbers come out the same.
"""
def __init__(self, d_model, d_ff, n_experts, k, n_shared=0):
super().__init__()
self.k = k
self.n_experts = n_experts
self.router = torch.nn.Linear(d_model, n_experts, bias=False)
self.experts = torch.nn.ModuleList(
torch.nn.Sequential(
torch.nn.Linear(d_model, d_ff, bias=False),
torch.nn.GELU(),
torch.nn.Linear(d_ff, d_model, bias=False),
)
for _ in range(n_experts)
)
# Shared experts always run. Routed experts compete. Mixtral has
# n_shared = 0. DeepSeekMoE adds one (or more) shared experts.
self.shared = torch.nn.ModuleList(
torch.nn.Sequential(
torch.nn.Linear(d_model, d_ff, bias=False),
torch.nn.GELU(),
torch.nn.Linear(d_ff, d_model, bias=False),
)
for _ in range(n_shared)
)
def forward(self, x):
# x: (T, d) for a packed batch of tokens. The residual lives outside.
logits = self.router(x)
probs = F.softmax(logits, dim=-1) # for the aux loss
gates = topk_route(logits, self.k) # (T, N)
out = torch.zeros_like(x)
for i, expert in enumerate(self.experts):
weight = gates[:, i].unsqueeze(-1)
# Tokens with weight 0 still run the expert here. A real kernel
# would skip them. We keep the loop honest and short.
out = out + weight * expert(x)
for expert in self.shared:
out = out + expert(x)
return out, switch_aux_loss(probs, gates, self.n_experts)
def expert_capacity(n_tokens, k, n_experts, capacity_factor):
"""How many tokens one expert is allowed this step.
C = (T * k / N) * CF. CF > 1 leaves slack. Tokens over capacity drop.
"""
return (n_tokens * k / n_experts) * capacity_factor
Softmax first, then top-k, and stop. The kept weights will not sum to 1. Softmax again on the keepers (or mask with \(-\infty\) and softmax once).
Inverting capacity. CF > 1 is slack, not a shrink. \(C = (T k / N) \cdot \mathrm{CF}\).
Calling Mixtral's experts SiLU. They are SwiGLU, like Mistral 7B and LLaMA.
Counting 8 × 7B = 56B. Attention and embeddings are not copied. Mixtral is 46.7B.
E1 is off. Gates 0.731 and 0.269 on E0 and E2.
Each token picks \(k\) experts, so \(f_i = k/N\). Then \(N \sum (k/N)(1/N) = k\).
\((8\cdot 2 / 4)\cdot 1.25 = 5\).