DECODING DEEPSEEK · CHAPTER 04
A giant sparse feed-forward: one expert everyone shares, 256 small specialists, and a bias that balances load without a fat auxiliary loss.
Deep Cuts 07 is vanilla MoE: replace the FFN with \(N\) copies, pick top-k, add a loss so nobody starves. Recap in one line: Mixtral-style models do this with a handful of large experts.
DeepSeek’s kitchen is different. One pantry chef (the shared expert) tastes every plate, so the stations do not each keep a sack of salt. The stations themselves are small and many, so a token can mix “citrus + char + broth” instead of being stuck with two giant house sauces. And instead of yelling at the staff with a large extra loss, a tiny bias on the booking sheet nudges busy stations off the board.
V3: 671B total parameters, 37B active per token. The first of 61 layers is dense. Every later layer is this MoE. A token always runs 1 shared expert plus 8 of 256 routed experts. Not top-6. Top-6 is V2.
Take 8 experts of width \(4w\), pick top-2. Or split each into four: 32 experts of width \(w\), pick top-8. Parameters match (\(8 \times 4w = 32w\)). FLOPs match (2 fat experts ≈ 8 thin ones). Combinations do not: 28 mixes become 10,518,300.
same compute, many more ways to mix knowledge. That is the point of “fine-grained”: when you multiply the expert count by m, you also multiply top-k by m.
V3’s 256 routed experts at top-8 are this idea at production size. If you split and then kept top-2, you would have changed the compute. That misses the design.
Without a shared expert, every specialist relearns punctuation, English syntax, residual-stream hygiene. That is wasted capacity. V3 keeps one shared SwiGLU that every token runs, ungated, then adds the routed mix:
shared expert always; eight routed experts, weighted by gates that sum to one.
V2 used two shared experts and top-6 of 160. V3 moved to one shared and top-8 of 256. Do not mix those configs.
A large auxiliary loss fights the main loss. DeepSeek-V3 almost drops it. Each routed expert gets a bias \(b_i\), updated from how many tokens it just got:
Affinity is a sigmoid of the router logit. V3 switched from V2’s softmax.
Pick top-8 of \(\text{affinity} + b\). The bias is a booking nudge. It is not part of the gate.
Gates are the chosen affinities, renormalised so they sum to 1. Overloaded experts can be skipped next step without shrinking their historical gate quality.
After the step, \(b_i \leftarrow b_i + \gamma\,\operatorname{sign}(\bar n - n_i)\). Too many tokens: bias drops. \(\gamma = 0.001\), then 0 once things settle. A tiny sequence-wise balance loss (\(\alpha = 0.0001\)) stays as a safety net.
affinity · +b · update
Interactive · Bias as a thermostat
Four experts, one busy. What does one bias update do to next step’s chances?
V3 also does not drop tokens. Capacity-factor dropping is a vanilla-MoE knob; DeepSeek refused it. Communication is capped instead: a token’s 8 experts may live on at most 4 nodes (node-limited routing).
The bias is a buffer, not an nn.Parameter. The check script confirms an overloaded expert’s bias drops, and that cranking the bias does not change the stored affinities.
"""DeepSeekMoE: shared expert, fine-grained routed experts, loss-free bias.
Vanilla MoE (top-k of N experts plus a balancing loss) is taught in
Deep Cuts. This file is only what DeepSeek changed:
* one shared expert that every token always runs
* many small routed experts instead of a few large ones
* a bias on the router used for selection, not for the gate weights
* sigmoid affinities, then renormalise the chosen ones
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
class SwiGLUExpert(nn.Module):
"""One small SwiGLU feed-forward. DeepSeek's experts use this, not GELU."""
def __init__(self, d_model, d_ff):
super().__init__()
self.w_gate = nn.Linear(d_model, d_ff, bias=False)
self.w_up = nn.Linear(d_model, d_ff, bias=False)
self.w_down = nn.Linear(d_ff, d_model, bias=False)
def forward(self, x):
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
class LossFreeRouter(nn.Module):
"""Sigmoid affinities. Bias is added only when picking the top-k."""
def __init__(self, d_model, n_routed, top_k, gamma=0.001):
super().__init__()
self.top_k = top_k
self.gamma = gamma
self.w = nn.Linear(d_model, n_routed, bias=False)
# Not a trained parameter. Updated from token counts after each step.
self.register_buffer("bias", torch.zeros(n_routed))
def forward(self, x):
# Affinity is a sigmoid, not a softmax. V3 made this switch from V2.
affinity = torch.sigmoid(self.w(x)) # (B, T, E)
# Bias steers *who is chosen*. It does not change the gate values.
_, idx = torch.topk(affinity + self.bias, self.top_k, dim=-1)
chosen = torch.gather(affinity, -1, idx)
gates = chosen / chosen.sum(dim=-1, keepdim=True)
return gates, idx, affinity
@torch.no_grad()
def update_bias(self, idx, n_routed):
"""Push overloaded experts down and idle experts up.
violation = mean_load - load. An expert that got too many tokens
has a negative violation, so its bias drops and it is chosen less
often next step. DeepSeek-V3 uses gamma = 0.001, then 0.
"""
flat = idx.reshape(-1)
load = torch.bincount(flat, minlength=n_routed).float()
violation = load.mean() - load
self.bias += self.gamma * torch.sign(violation)
class DeepSeekMoE(nn.Module):
"""Shared expert always runs; top-k of the routed experts mix in."""
def __init__(self, d_model, d_ff, n_routed, top_k, gamma=0.001):
super().__init__()
self.n_routed = n_routed
self.top_k = top_k
self.shared = SwiGLUExpert(d_model, d_ff)
self.routed = nn.ModuleList(
[SwiGLUExpert(d_model, d_ff) for _ in range(n_routed)]
)
self.router = LossFreeRouter(d_model, n_routed, top_k, gamma)
def forward(self, x, update_bias=True):
gates, idx, _ = self.router(x)
# Shared expert: every token, no gate. This is the "common knowledge"
# path so routed experts do not all have to relearn the same basics.
shared = self.shared(x)
routed = torch.zeros_like(x)
for slot in range(self.top_k):
slot_idx = idx[:, :, slot]
slot_gate = gates[:, :, slot]
for e, expert in enumerate(self.routed):
mask = slot_idx == e
if not mask.any():
continue
expert_out = expert(x[mask])
routed[mask] = routed[mask] + slot_gate[mask].unsqueeze(-1) * expert_out
if update_bias and self.training:
self.router.update_bias(idx, self.n_routed)
return shared + routed
gamma=0.001 is V3’s published start value.“V3 is top-6.” V2 is top-6 of 160 plus 2 shared. V3 is top-8 of 256 plus 1 shared.
Fine-grained without raising top-k. Split \(m\) ways and pick \(mK\), or you changed the FLOPs and missed the combination gain.
Adding the bias into the gate softmax/sigmoid. Selection uses affinity + bias. The gate weights are the raw affinities of whoever was selected.
“The first three layers are dense, so the 37B is a lie.” This series follows the published box: 61 layers, the first is dense, 37B active. Do not invent a second count.
Nine: 1 shared + 8 routed. Never “top-6”.
It goes from 2 to 8.
It decreases (\(\operatorname{sign}(\bar n - n_0) = -1\)), so expert 0 is less likely to be selected next step. Its gate formula does not change.
No. It balances with the bias (and a tiny sequence loss) and caps the number of nodes a token may touch.