Mixture of Experts: Sparse Compute, Dense Knowledge

Mixture of Experts (MoE) replaces the FFN with a set of expert FFNs and a learned router. For each token, the router picks the top-K experts (K=1 or K=2 typically) and only those experts compute. Total parameters are large; active parameters per token are much smaller.

This is the architectural shape of choice for high-quality models that need to be cheap to serve. You get a 100B-parameter model's knowledge with a 20B-parameter model's per-token compute.

Model	Total params	Active params	Experts	Routing
Mixtral 8×7B	47B	13B	8	top-2
Mixtral 8×22B	141B	39B	8	top-2
Llama 4 Scout	109B	17B	16 (routed)	top-1
Llama 4 Maverick	400B	17B	128 + 1 shared	top-1
DeepSeek-V3	671B	37B	128 + 1 shared	top-K
Mistral Small 4	119B	6B	128	top-4

Why MoE works

Different tokens need different kinds of "thinking." A code token benefits from one expert; a Korean token benefits from another; a numerical reasoning step benefits from a third. Letting the router specialize experts allows the model to use its parameters efficiently — only the right ones fire for each token. The cost: training is harder (load balancing, routing collapse), serving requires custom inference frameworks, and per-batch tail latency is variable.

Code

Top-2 MoE block (sketch)·python

class MoELayer(nn.Module):
    def __init__(self, d_model, d_ff, n_experts, top_k=2):
        super().__init__()
        self.experts = nn.ModuleList(
            [SwiGLU(d_model, d_ff) for _ in range(n_experts)]
        )
        self.router = nn.Linear(d_model, n_experts, bias=False)
        self.top_k = top_k
    def forward(self, x):
        # x: (B, L, d_model)
        logits = self.router(x)                            # (B, L, n_experts)
        topk_vals, topk_idx = logits.topk(self.top_k, dim=-1)
        gates = F.softmax(topk_vals, dim=-1)              # normalize over chosen
        out = torch.zeros_like(x)
        for k in range(self.top_k):
            for e in range(len(self.experts)):
                mask = (topk_idx[..., k] == e)
                if mask.any():
                    out[mask] += gates[..., k][mask].unsqueeze(-1) * self.experts[e](x[mask])
        return out
# Real implementations dispatch tokens to experts in batched form
# (custom kernel territory) for performance.

Exercise

Conceptually compare three approaches to scaling: (a) dense Llama 3 70B, (b) Mixtral 8x22B (141B total / 39B active), (c) Llama 4 Maverick (400B / 17B active). For each, estimate parameter memory at FP16, KV cache at 32K context, and approximate per-token throughput. Which is cheapest to serve at high quality?

Mixture of Experts: Sparse Compute, Dense Knowledge

Why MoE works

Code

External links

Exercise

Progress

Comments 0