THE TRANSFORMER DEEP CUTS · CHAPTER 07

Mixture of Experts

Replace one FFN with many, and let a router pick a few per token. Most of the extra parameters stay asleep.

~18 min read1 interactiveTested PyTorch code

What you'll learn

  • Top-k routing: logits, a hard pick, then softmax on the keepers
  • Why a load-balancing loss exists, and the Switch formula \(N \sum f_i P_i\)
  • Capacity: \(C = (T \cdot k / N) \cdot \mathrm{CF}\). CF > 1 grows the bucket
  • Shared experts always run; routed experts compete

Before you start

A panel, not a soloist

A harbour radio has four specialists on call: weather, traffic, repairs, stores. Each incoming phrase is handed to two of them. The desk mixes their two replies. The other two specialists sit this one out.

That is sparse MoE. The specialists are copies of the FFN. The desk is a tiny linear router. Attention is unchanged and still runs for every token.

DeepSeekMoE — fine-grained experts, a shared expert, loss-free bias — is a later design. It has its own series. This chapter is the generic router those papers start from.

Pick, then softmax

For each token the router writes \(N\) logits. Keep the top \(k\). Fill the rest with \(-\infty\). Softmax what remains. The output is a weighted sum of the \(k\) chosen FFNs, and it has the same shape as a dense FFN, so the residual still fits.

Token rain, three experts, top-2, logits \([2.0,\, 0.0,\, 1.0]\):

E0E1E2
logit2.00.01.0
after top-2 softmax0.73100.269

E1 is dropped. The mix is \(0.731\, E_0(x) + 0.269\, E_2(x)\).

Interactive · A three-expert desk

Move the three logits. Which two experts stay on, and at what weight?

2.0
0.0
1.0

Stop the rich from getting richer

A router that already likes expert 0 sends that expert more tokens, so it gets more gradient, so the router likes it more. Collapse. The Switch Transformer adds

\(f_i\) is the fraction of tokens that actually picked expert \(i\) (hard). \(P_i\) is the mean full-softmax probability of expert \(i\) (soft). The product is smallest when both are flat.

Eight tokens, four experts, top-2. Balanced: each expert is picked by half the tokens, \(P_i = 1/4\), and \(L_{\mathrm{aux}} = 2\) — equal to \(k\). Collapsed onto experts 0 and 1: \(L_{\mathrm{aux}} = 4\). The extra loss pushes the router back toward even traffic.

\(P_i\) is taken from the full softmax, not from the top-k gates. That is the Switch paper. Using the gates instead misses the gradient that would have gone to a dropped expert.

balanced \(L=2\)  →  collapse  →  \(L=4\)  →  push flat

MANIM Balanced traffic → \(L_{\mathrm{aux}}=k\); collapse doubles it.

A bucket per expert

expected load, times a slack factor. CF > 1 makes the bucket larger. Tokens over capacity are dropped (or rerouted).

Eight tokens, top-2, four experts, CF \(= 1.25\): \(C = (8 \cdot 2 / 4) \cdot 1.25 = 5\). An older write-up inverted this and made CF > 1 shrink the bucket. That is the wrong way around.

T·k/N  →  CF=1.25  →  C=5  →  overflow

MANIM CF > 1 grows the bucket; overflow drops.

Someone always shows up

A routed expert competes for tokens. A shared expert always runs, so common work ("how English works") does not have to be copied into every specialist. Mixtral 8×7B has no shared expert: 8 experts, top-2, all SwiGLU. About 46.7B total parameters, about 12.9B active. That is not \(8 \times 7.24\)B, because attention, embeddings, and norms are shared.

DeepSeek-V3, as a pointer only: 256 routed + 1 shared, top-8, 671B total, 37B active. The extra ideas (fine-grained split, loss-free bias, device-limited routing) stay in Decoding DeepSeek.

A teaching loop

The Python walks every expert, even the ones with weight 0. A production kernel gathers the tokens of each expert instead. The numbers match. Mixtral's experts are SwiGLU; the file uses GELU so it does not depend on chapter 05.

moe.pytop-k gates [0.731, 0, 0.269]; capacity 5; aux = k when balanced
"""A generic top-k mixture of experts. Not DeepSeekMoE.

DeepSeek's fine-grained experts, shared-expert layout, and loss-free
bias live in Decoding DeepSeek. This file is the idea those papers start from:
N copies of an FFN, a router, top-k, and a load-balancing loss.
"""
import torch
import torch.nn.functional as F


def topk_route(logits, k):
    """logits: (..., n_experts) -> gates (same shape, zeros on the unpicked).

    Softmax runs on the kept experts only. A dropped expert gets weight 0
    and no gradient through the softmax (its logit was never selected).
    """
    values, index = torch.topk(logits, k, dim=-1)
    # -inf on the unpicked slots, then softmax over experts.
    masked = torch.full_like(logits, float("-inf")).scatter(-1, index, values)
    return F.softmax(masked, dim=-1)


def switch_aux_loss(router_probs, gates, n_experts):
    """Switch-Transformer load-balancing loss: N * sum_i f_i P_i.

    P_i is the mean *full* softmax probability of expert i (soft).
    f_i is the fraction of tokens that actually picked expert i (hard).
    The product is smallest when every expert is equally popular.
    """
    # router_probs: (T, N) full softmax. gates: (T, N) after top-k.
    p = router_probs.mean(dim=0)                               # (N,)
    # A token "picked" an expert when that expert's gate is > 0.
    f = (gates > 0).float().mean(dim=0)                        # (N,)
    return n_experts * (f * p).sum()


class MoE(torch.nn.Module):
    """N SwiGLU-shaped experts (linear + GELU + linear) and a top-k router.

    The teaching loop walks every expert. A production kernel batches the
    tokens of each expert instead. The numbers come out the same.
    """

    def __init__(self, d_model, d_ff, n_experts, k, n_shared=0):
        super().__init__()
        self.k = k
        self.n_experts = n_experts
        self.router = torch.nn.Linear(d_model, n_experts, bias=False)
        self.experts = torch.nn.ModuleList(
            torch.nn.Sequential(
                torch.nn.Linear(d_model, d_ff, bias=False),
                torch.nn.GELU(),
                torch.nn.Linear(d_ff, d_model, bias=False),
            )
            for _ in range(n_experts)
        )
        # Shared experts always run. Routed experts compete. Mixtral has
        # n_shared = 0. DeepSeekMoE adds one (or more) shared experts.
        self.shared = torch.nn.ModuleList(
            torch.nn.Sequential(
                torch.nn.Linear(d_model, d_ff, bias=False),
                torch.nn.GELU(),
                torch.nn.Linear(d_ff, d_model, bias=False),
            )
            for _ in range(n_shared)
        )

    def forward(self, x):
        # x: (T, d) for a packed batch of tokens. The residual lives outside.
        logits = self.router(x)
        probs = F.softmax(logits, dim=-1)                      # for the aux loss
        gates = topk_route(logits, self.k)                     # (T, N)
        out = torch.zeros_like(x)
        for i, expert in enumerate(self.experts):
            weight = gates[:, i].unsqueeze(-1)
            # Tokens with weight 0 still run the expert here. A real kernel
            # would skip them. We keep the loop honest and short.
            out = out + weight * expert(x)
        for expert in self.shared:
            out = out + expert(x)
        return out, switch_aux_loss(probs, gates, self.n_experts)


def expert_capacity(n_tokens, k, n_experts, capacity_factor):
    """How many tokens one expert is allowed this step.

    C = (T * k / N) * CF. CF > 1 leaves slack. Tokens over capacity drop.
    """
    return (n_tokens * k / n_experts) * capacity_factor

Mistakes learners make

Softmax first, then top-k, and stop. The kept weights will not sum to 1. Softmax again on the keepers (or mask with \(-\infty\) and softmax once).

Inverting capacity. CF > 1 is slack, not a shrink. \(C = (T k / N) \cdot \mathrm{CF}\).

Calling Mixtral's experts SiLU. They are SwiGLU, like Mistral 7B and LLaMA.

Counting 8 × 7B = 56B. Attention and embeddings are not copied. Mixtral is 46.7B.

What to remember

  • MoE replaces the FFN. Attention is still dense.
  • Top-k, then softmax on the keepers. Same output shape, residual still works.
  • Balance with \(N \sum f_i P_i\). Capacity is \((T k / N)\cdot\mathrm{CF}\).
  • Shared experts always run. Mixtral has none; DeepSeekMoE does.

Check yourself

1. Logits [2, 0, 1], top-2. Which expert is off, and what are the two gates?

E1 is off. Gates 0.731 and 0.269 on E0 and E2.

2. Why is the balanced aux loss equal to \(k\), not 1?

Each token picks \(k\) experts, so \(f_i = k/N\). Then \(N \sum (k/N)(1/N) = k\).

3. Eight tokens, top-2, four experts, CF 1.25. Capacity?

\((8\cdot 2 / 4)\cdot 1.25 = 5\).

Go to the source

Keep going