DECODING DEEPSEEK · CHAPTER 04

DeepSeekMoE

A giant sparse feed-forward: one expert everyone shares, 256 small specialists, and a bias that balances load without a fat auxiliary loss.

~18 min read1 interactiveTested PyTorch code

What you'll learn

  • Why DeepSeek splits experts finer and always activates more of them
  • What the shared expert is for
  • How a bias on the router replaces a large balancing loss
  • The V3 config: 256 routed + 1 shared, top-8, 671B / 37B

Before you start

A kitchen with a pantry chef and many stations

Deep Cuts 07 is vanilla MoE: replace the FFN with \(N\) copies, pick top-k, add a loss so nobody starves. Recap in one line: Mixtral-style models do this with a handful of large experts.

DeepSeek’s kitchen is different. One pantry chef (the shared expert) tastes every plate, so the stations do not each keep a sack of salt. The stations themselves are small and many, so a token can mix “citrus + char + broth” instead of being stuck with two giant house sauces. And instead of yelling at the staff with a large extra loss, a tiny bias on the booking sheet nudges busy stations off the board.

V3: 671B total parameters, 37B active per token. The first of 61 layers is dense. Every later layer is this MoE. A token always runs 1 shared expert plus 8 of 256 routed experts. Not top-6. Top-6 is V2.

Split the cooks, keep the heat

Take 8 experts of width \(4w\), pick top-2. Or split each into four: 32 experts of width \(w\), pick top-8. Parameters match (\(8 \times 4w = 32w\)). FLOPs match (2 fat experts ≈ 8 thin ones). Combinations do not: 28 mixes become 10,518,300.

same compute, many more ways to mix knowledge. That is the point of “fine-grained”: when you multiply the expert count by m, you also multiply top-k by m.

V3’s 256 routed experts at top-8 are this idea at production size. If you split and then kept top-2, you would have changed the compute. That misses the design.

Common knowledge, always on

Without a shared expert, every specialist relearns punctuation, English syntax, residual-stream hygiene. That is wasted capacity. V3 keeps one shared SwiGLU that every token runs, ungated, then adds the routed mix:

shared expert always; eight routed experts, weighted by gates that sum to one.

V2 used two shared experts and top-6 of 160. V3 moved to one shared and top-8 of 256. Do not mix those configs.

routed · shared · V3

MANIM Routed compete; shared always runs.

Bias for selection, not for the gate

A large auxiliary loss fights the main loss. DeepSeek-V3 almost drops it. Each routed expert gets a bias \(b_i\), updated from how many tokens it just got:

  1. Affinity is a sigmoid of the router logit. V3 switched from V2’s softmax.

  2. Pick top-8 of \(\text{affinity} + b\). The bias is a booking nudge. It is not part of the gate.

  3. Gates are the chosen affinities, renormalised so they sum to 1. Overloaded experts can be skipped next step without shrinking their historical gate quality.

  4. After the step, \(b_i \leftarrow b_i + \gamma\,\operatorname{sign}(\bar n - n_i)\). Too many tokens: bias drops. \(\gamma = 0.001\), then 0 once things settle. A tiny sequence-wise balance loss (\(\alpha = 0.0001\)) stays as a safety net.

affinity · +b · update

MANIM Bias nudges selection; gates stay pure affinities.

Interactive · Bias as a thermostat

Four experts, one busy. What does one bias update do to next step’s chances?

V3 also does not drop tokens. Capacity-factor dropping is a vanilla-MoE knob; DeepSeek refused it. Communication is capped instead: a token’s 8 experts may live on at most 4 nodes (node-limited routing).

Shared + routed + a buffer bias

The bias is a buffer, not an nn.Parameter. The check script confirms an overloaded expert’s bias drops, and that cranking the bias does not change the stored affinities.

deepseek_moe.pyshape preserved; bias for selection only; shared expert always runs
"""DeepSeekMoE: shared expert, fine-grained routed experts, loss-free bias.

Vanilla MoE (top-k of N experts plus a balancing loss) is taught in
Deep Cuts. This file is only what DeepSeek changed:

  * one shared expert that every token always runs
  * many small routed experts instead of a few large ones
  * a bias on the router used for selection, not for the gate weights
  * sigmoid affinities, then renormalise the chosen ones
"""
import torch
import torch.nn as nn
import torch.nn.functional as F


class SwiGLUExpert(nn.Module):
    """One small SwiGLU feed-forward. DeepSeek's experts use this, not GELU."""

    def __init__(self, d_model, d_ff):
        super().__init__()
        self.w_gate = nn.Linear(d_model, d_ff, bias=False)
        self.w_up = nn.Linear(d_model, d_ff, bias=False)
        self.w_down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))


class LossFreeRouter(nn.Module):
    """Sigmoid affinities. Bias is added only when picking the top-k."""

    def __init__(self, d_model, n_routed, top_k, gamma=0.001):
        super().__init__()
        self.top_k = top_k
        self.gamma = gamma
        self.w = nn.Linear(d_model, n_routed, bias=False)
        # Not a trained parameter. Updated from token counts after each step.
        self.register_buffer("bias", torch.zeros(n_routed))

    def forward(self, x):
        # Affinity is a sigmoid, not a softmax. V3 made this switch from V2.
        affinity = torch.sigmoid(self.w(x))           # (B, T, E)
        # Bias steers *who is chosen*. It does not change the gate values.
        _, idx = torch.topk(affinity + self.bias, self.top_k, dim=-1)
        chosen = torch.gather(affinity, -1, idx)
        gates = chosen / chosen.sum(dim=-1, keepdim=True)
        return gates, idx, affinity

    @torch.no_grad()
    def update_bias(self, idx, n_routed):
        """Push overloaded experts down and idle experts up.

        violation = mean_load - load. An expert that got too many tokens
        has a negative violation, so its bias drops and it is chosen less
        often next step. DeepSeek-V3 uses gamma = 0.001, then 0.
        """
        flat = idx.reshape(-1)
        load = torch.bincount(flat, minlength=n_routed).float()
        violation = load.mean() - load
        self.bias += self.gamma * torch.sign(violation)


class DeepSeekMoE(nn.Module):
    """Shared expert always runs; top-k of the routed experts mix in."""

    def __init__(self, d_model, d_ff, n_routed, top_k, gamma=0.001):
        super().__init__()
        self.n_routed = n_routed
        self.top_k = top_k
        self.shared = SwiGLUExpert(d_model, d_ff)
        self.routed = nn.ModuleList(
            [SwiGLUExpert(d_model, d_ff) for _ in range(n_routed)]
        )
        self.router = LossFreeRouter(d_model, n_routed, top_k, gamma)

    def forward(self, x, update_bias=True):
        gates, idx, _ = self.router(x)
        # Shared expert: every token, no gate. This is the "common knowledge"
        # path so routed experts do not all have to relearn the same basics.
        shared = self.shared(x)
        routed = torch.zeros_like(x)
        for slot in range(self.top_k):
            slot_idx = idx[:, :, slot]
            slot_gate = gates[:, :, slot]
            for e, expert in enumerate(self.routed):
                mask = slot_idx == e
                if not mask.any():
                    continue
                expert_out = expert(x[mask])
                routed[mask] = routed[mask] + slot_gate[mask].unsqueeze(-1) * expert_out
        if update_bias and self.training:
            self.router.update_bias(idx, self.n_routed)
        return shared + routed
  • This loop is pedagogical. A real kernel dispatches tokens to experts, it does not run every expert on a mask.
  • gamma=0.001 is V3’s published start value.
  • SwiGLU, not GELU: that is the expert body in V3.

Mistakes learners make

“V3 is top-6.” V2 is top-6 of 160 plus 2 shared. V3 is top-8 of 256 plus 1 shared.

Fine-grained without raising top-k. Split \(m\) ways and pick \(mK\), or you changed the FLOPs and missed the combination gain.

Adding the bias into the gate softmax/sigmoid. Selection uses affinity + bias. The gate weights are the raw affinities of whoever was selected.

“The first three layers are dense, so the 37B is a lie.” This series follows the published box: 61 layers, the first is dense, 37B active. Do not invent a second count.

What to remember

  • Fine-grained experts: more of them, each smaller, top-k scaled with the split. Same FLOPs, many more mixes (\(\binom{32}{8} = 10{,}518{,}300\) vs \(\binom{8}{2} = 28\) in the toy).
  • One shared expert, always on, so routed experts do not store the same basics.
  • Loss-free bias: \(\gamma = 0.001\), then 0. Bias for who is picked, not for how much they weigh.
  • V3: 256 routed + 1 shared, top-8, 671B total, 37B active.

Check yourself

1. A V3 token uses how many experts?

Nine: 1 shared + 8 routed. Never “top-6”.

2. You split 8 experts into 32. What must happen to top-k to keep FLOPs constant?

It goes from 2 to 8.

3. Expert 0 got twice the mean load. What happens to \(b_0\)?

It decreases (\(\operatorname{sign}(\bar n - n_0) = -1\)), so expert 0 is less likely to be selected next step. Its gate formula does not change.

4. Does V3 drop overflow tokens?

No. It balances with the bias (and a tiny sequence loss) and caps the number of nodes a token may touch.

Go to the source

Keep going