Mixture of Experts Changes the Arithmetic

"Capacity follows total parameters. Speed follows active ones. The runtime decides how much of the second promise you get."

Two Numbers Instead of One

A dense model has one parameter count, and it is both the memory it needs and the bytes each token reads. A mixture-of-experts model splits every feed-forward layer into many experts and routes each token through a few of them — 8 of 256 in Qwen3.5-35B-A3B, 8 of 288 in GLM-5.3-Flash, 6 of 384 in DeepSeek-V4.1-Flash, 16 of 896 in Kimi K3. So it has two counts: total parameters, which set the memory footprint because every expert must be resident, and active parameters, which set the bytes per token because only the routed experts are read. The names say it: 35B total, 3B active.

For the decode ceiling this is a large change. The lab's bytes-per-token accounting reads the 35B-A3B's 18.1 GB of expert tensors at 8/256 and adds the shared expert, attention and head that every token touches: 1.66 GB per token from a 20.4 GB file. Its ceiling on office is therefore near 385 tokens per second — between the 4B's 269 and the 2B's 602 — while its capacity is that of a 27B. The same arithmetic on the household's larger checkpoints:

Checkpoint	Total / active (vendor card)	Experts, routed per token	Footprint on disk	Bytes per token (derived)	Evidence
Qwen3.5-35B-A3B (4-bit)	35B / 3B	256, 8 (+1 shared)	20.39 GB	1.66 GB	physics from the headers
Qwen3.8-Flash-Next (bf16)	125B + 51B n-gram + 4B MTP / 6B	512, 10	360.0 GB	~12 GB at bf16; ~3 GB at 4-bit	vendor card + derived
GLM-5.3-Flash (bf16)	320B / 18B	288, 8	642.7 GB	~36 GB at bf16; ~9 GB at 4-bit	vendor card + derived
DeepSeek-V4.1-Flash (FP4/FP8 mixed)	552B + 196B memory / 8B prefill, 16B decode	384, 6	510.3 GB	~8 GB at the shipped mixed precision	vendor card + derived
GLM-5.3 (bf16)	744B / 40B	256, 8 (+1 shared)	1,506.7 GB	~80 GB at bf16; ~20 GB at 4-bit	vendor card + derived
Kimi K3 (MXFP4 experts)	2.8T / 104B	896, 16 (+2 shared)	1,561.0 GB	~52 GB at the shipped precision	vendor card + derived

Read the last two columns together. A 744-billion-parameter model that needs a 512 GB Mac just to load at 4 bits would, by the arithmetic, decode at 638 ÷ 20 ≈ 32 tokens per second on office — the same speed as a dense 27B. That is the promise of the architecture, and it is why the largest open checkpoints of 2026 are all mixtures: the memory ceiling and the bandwidth ceiling are different numbers, and MoE lets a model sit at the top of the first while staying near the bottom of the second.

What the Runtime Does With the Promise

Now the measurement. The 35B-A3B's ceiling is 385 tokens per second on office, 236 on pro2023, 442 on music. Measured: 89, 110, 101.5. Twenty-three to forty-seven per cent of the ceiling, against 67–74% for the dense 9B and 27B on the same machines — and, strangest of all, fastest on the M3 Max, the machine with the least bandwidth of the three. The bytes are not the bottleneck. A batch-1 decode step of this model dispatches, in every one of its 40 layers, a routed gather across the selected experts and a set of small matrix-vector kernels — many launches, each too small to fill eighty GPU cores, each paying the per-token fixed overhead the lab track measured. The Max's lower overhead per token (1.1 ms against 1.7–1.8 on the Ultras) is worth more here than the Ultras' extra bandwidth. Active-parameter arithmetic sets the ceiling; per-expert kernel dispatch sets the floor; and for a small MoE at batch 1 on mlx-lm 0.31, the floor is where you live.

This is not a verdict on the architecture. Prefill on the same model runs at 1,276 tokens per second on office — faster than the dense 9B — because prefill batches many tokens through the same experts and the kernels fill. Serving many users at once (next lesson) has the same effect on decode. And a better runtime can close the gap: the dispatch cost is software. But the honest label for "3B active, so it decodes like a 3B" is ceiling, not measurement, and this quest's measurement is the label.

Code

moe_promise.py — the ceiling MoE promises against what the ladder measured·python

#!/usr/bin/env python3
"""For each lab Mac: the 35B-A3B's ceiling from achieved bandwidth and its
1.659 GB per token, the measured decode rate, and the fraction reached —
beside the dense 9B on the same machine for contrast."""

ACHIEVED = {"office": 638, "pro2023": 391, "music": 734}      # GB/s, stream.py sum (read-only)
BYTES = {"Qwen3.5-35B-A3B-4bit": 1.659, "Qwen3.5-9B-4bit": 4.466}
MEASURED = {                                                    # decode tok/s, 2026-09-15
    "office":  {"Qwen3.5-35B-A3B-4bit": 89.1,  "Qwen3.5-9B-4bit": 95.1},
    "pro2023": {"Qwen3.5-35B-A3B-4bit": 109.9, "Qwen3.5-9B-4bit": 72.9},
    "music":   {"Qwen3.5-35B-A3B-4bit": 101.5, "Qwen3.5-9B-4bit": 105.5},
}

print(f"{'alias':8} {'model':22} {'ceiling':>8} {'measured':>9} {'fraction':>9}")
for alias, bw in ACHIEVED.items():
    for model, gb in BYTES.items():
        ceiling = bw / gb
        meas = MEASURED[alias][model]
        print(f"{alias:8} {model:22} {ceiling:8.0f} {meas:9.1f} {meas/ceiling:9.0%}")

print("\nThe MoE reaches a fifth to a half of its ceiling; the dense 9B two thirds.")
print("The MoE is fastest on the machine with the least bandwidth and the least per-token overhead.")

Exercise

Run moe_promise.py, then repeat the measurement on your own Mac with the 35B-A3B and a dense model of similar bytes-per-token (the 2B is the nearest on the ladder). Record both ceilings and both measured rates. Then answer: which of the two would you deploy for a batch-1 chat assistant on your machine, and which for a nightly batch job over ten thousand documents?

Hint

At batch 1 the dense 2B likely decodes as fast or faster and is a fifth of the footprint; the MoE's extra capacity buys quality per token you would have to judge, not score. For the batch job the MoE's ceiling starts to matter, because batched decode fills the expert kernels — the next lesson's subject.

Two Numbers Instead of One

What the Runtime Does With the Promise

Code

External links

Exercise

Progress

Comments 0