Routers, Top-K, and Shared Experts

The router is the brain of MoE

The router is a tiny network — usually a single linear layer mapping the token's hidden state to a logits vector over experts — but it is responsible for the entire routing decision. The router is learned alongside the rest of the model. If the router is bad, the experts get useless training signal and everything falls apart.

How top-K works

For each token, the router emits N scores (one per expert). Top-K selection keeps the K highest scores; the rest are zeroed out. The kept scores are passed through softmax (or sigmoid for some 2025+ designs) to become routing weights. Each selected expert's output is multiplied by its weight, and the K weighted outputs are summed.

Common K values in practice

Top-1: Switch Transformer's original choice. Cheapest but most prone to expert collapse. Llama 4 Scout uses top-1.
Top-2: Mixtral's choice. Two experts hedge against routing mistakes; modest extra compute. Was the dominant choice in 2023–2024.
Top-6 to Top-8: DeepSeek's fine-grained-expert designs. Many small experts, more routes selected per token, more nuanced specialization.

Shared experts

DeepSeek-V2/V3 and Gemma 4 MoE include 1–2 shared experts that always activate for every token, regardless of router decisions. They provide a stable always-on baseline so the routed experts don't have to relearn common patterns. This is a small architectural addition with big training-stability benefits.

Sigmoid vs softmax routing

Most early MoE used softmax over expert logits. DeepSeek-V3 introduced sigmoid routing with auxiliary-loss-free balancing — each expert score is independent (no zero-sum competition), and load balancing is achieved by adding learned bias terms to the logits instead of an explicit auxiliary loss. This allowed DeepSeek-V3 to drop the auxiliary loss entirely and still avoid expert collapse.

Code

Router with top-K and softmax weighting·python

import torch
import torch.nn as nn
import torch.nn.functional as F

class TopKRouter(nn.Module):
    def __init__(self, d_model, num_experts, k=2):
        super().__init__()
        self.gate = nn.Linear(d_model, num_experts, bias=False)
        self.k = k

    def forward(self, x):
        logits = self.gate(x)                  # (..., num_experts)
        topk_vals, topk_idx = logits.topk(self.k, dim=-1)
        weights = F.softmax(topk_vals, dim=-1)
        return topk_idx, weights

DeepSeek-V3 style sigmoid routing with bias terms·python

class SigmoidRouter(nn.Module):
    def __init__(self, d_model, num_experts, k=8):
        super().__init__()
        self.gate = nn.Linear(d_model, num_experts, bias=False)
        # Learned per-expert bias for load balancing (no aux loss needed).
        self.expert_bias = nn.Parameter(torch.zeros(num_experts))
        self.k = k

    def forward(self, x):
        scores = torch.sigmoid(self.gate(x)) + self.expert_bias  # (..., num_experts)
        topk_vals, topk_idx = scores.topk(self.k, dim=-1)
        # Sigmoid scores are independent; no softmax over selected experts.
        return topk_idx, topk_vals

Exercise

Pick three MoE models — Mixtral 8x7B, DeepSeek-V3, Llama 4 Maverick — and write down their (N experts, top-K, shared experts) tuple. Notice how the design space has shifted from 'few large experts top-2' (Mixtral) to 'many small experts top-8 with shared' (DeepSeek). That shift is the single most important design-evolution story in MoE 2024–2026.