Total Params vs Active Params — The Number That Actually Matters

The number on the marketing slide is almost always wrong-shaped

When a model is announced as "200B parameters", that single number used to be enough. With MoE everywhere at frontier scale, you now need a pair of numbers — total parameters (everything in memory) and active parameters per token (everything that computes per token). The ratio between them controls almost every cost decision you make next.

The notation: 235B-A22B

The standard shorthand for MoE in 2025–2026 is XB-AYB — X total parameters, Y active per token. Qwen3 235B-A22B has 235B in memory but ~22B compute per token. DeepSeek-V3 671B-A37B has 671B in memory but ~37B compute per token. A single number ("70B") almost always means dense, where total = active.

Why this controls everything downstream

Memory: scales with total. You have to load all 671B weights of DeepSeek-V3 into the GPUs, even though only 37B fire per token. There is no way around this with current MoE serving.
FLOPs per token: scales with active. DeepSeek-V3 generates tokens at roughly the FLOP cost of a 37B dense model. That is what makes it serveable.
Quality ceiling: rises with total. The full 671B parameter capacity contributes to the model's knowledge, even though only 37B fire per token, because different experts fire on different tokens.

The mental model: a library, not a desk

Dense models are like a desk — every paper on the desk is in arm's reach for every question. MoE models are like a library — most books are on the shelves, only a few are pulled to the desk per question. The whole library still has to fit in the building.

Code

Total vs active in three real model classes·python

# Dense: total == active (every token activates everything)
llama_3_70b = {"total_B": 70, "active_B": 70}

# Mixtral 8x7B (early MoE, top-2 routing over 8 experts)
mixtral_8x7b = {"total_B": 46.7, "active_B": 12.9}

# DeepSeek-V3 (large MoE, top-8 over 256 experts + 1 shared)
deepseek_v3 = {"total_B": 671, "active_B": 37}

# Cost shapes
for name, m in [("L70B", llama_3_70b),
                ("Mixtral", mixtral_8x7b),
                ("V3", deepseek_v3)]:
    print(name, "memory ~", m["total_B"], "B  | FLOPs/tok ~", m["active_B"], "B")

Quick rule: serving cost approximations·python

# Memory needed (rough, BF16):
#   bytes ~= total_B * 1e9 * 2

# FLOPs per token (rough, prefill or decode):
#   flops_per_tok ~= 2 * active_B * 1e9

# Tokens/sec for a given hardware FLOP/s budget:
#   tps ~= hardware_flops / (2 * active_B * 1e9)

Exercise

Write a one-paragraph note to a non-technical PM at your company explaining why a '670B parameter model' might be cheaper to serve per token than a '70B parameter model', without using the words 'MoE' or 'expert'. The point is to internalize the explanation in plain language — if you cannot say it cleanly, you do not yet understand it.