The MoE Intuition — Not Every Token Needs Every Expert

The observation

Not every token in a sequence requires the same processing. The token "the" does not need the same computational pathway as a complex mathematical expression or a code identifier in a rare language. Dense models pay full price for every token regardless of difficulty. Mixture of Experts challenges that — what if only the parts of the network relevant to a token activated, and the rest stayed quiet?

The structural change in one sentence

MoE replaces the single dense FFN block in each Transformer layer with a small bank of expert FFNs plus a router. Everything else — attention, layer norms, embeddings, position encoding — is identical to a dense Transformer. The only place the architecture changes is inside the FFN.

The four ingredients

Experts. Independent feed-forward networks. Each expert has the same architecture but its own learned weights. Think of them as specialized sub-networks.
Router (gate). A small learned network — usually a linear layer + softmax or sigmoid — that produces a score over experts for each token.
Top-K selection. Only the K highest-scoring experts process the token. Common choices: top-2 (Mixtral), top-6 (DeepSeek-V2), top-8 (DeepSeek-V3, Qwen3 MoE).
Optional shared experts. Some architectures (DeepSeek) include 1–2 experts that always activate for every token. They provide a stable baseline pathway.

The promise

You get the capacity of a much larger model (because the total parameter pool is large) at roughly the per-token compute of a much smaller model (because only K of N experts fire). The catch — and it is a real catch — is that all N experts have to live in memory, even though only K fire per token. Memory ≠ compute, and MoE breaks that equivalence permanently.

What this lesson is not yet covering

How routing actually decides; the auxiliary load-balancing loss; expert collapse; the memory paradox; the differences between Mixtral, DeepSeek, Llama 4. All of those are coming. Right now the only goal is to internalize the shape: dense FFN → expert bank + router.

Code

Dense FFN vs MoE FFN — the only architectural diff·python

# Dense FFN: one network, all tokens use it.
def dense_ffn(x):
    return W_down @ silu(W_gate @ x) * (W_up @ x)   # all weights active

# MoE FFN: one router + N experts, top-K of them fire.
def moe_ffn(x, router, experts, k=2):
    scores  = router(x)                              # (N,)
    top_k   = topk_indices(scores, k)
    weights = softmax(scores[top_k])
    output  = 0
    for i, w in zip(top_k, weights):
        output += w * experts[i](x)                  # only k experts compute
    return output

Routing decision in BF16 with top-K masking·python

import torch

def route(x, gate_proj, k=2, num_experts=8):
    logits  = gate_proj(x)                  # (B*T, num_experts)
    weights, idx = logits.topk(k, dim=-1)   # top-K logits per token
    weights = weights.softmax(dim=-1)
    return idx, weights                     # idx: (B*T, k), weights: (B*T, k)

The MoE Intuition — Not Every Token Needs Every Expert

The observation

The structural change in one sentence

The four ingredients

The promise

What this lesson is not yet covering

Code

External links

Exercise

Progress

Comments 0