Prefill and Decode Across the M3 Ladder

"One microarchitecture, three sizes, one house. Double the dies and you do not double the tokens — and the two stages fall short by different amounts, which is the lesson."

Why One Generation Is the Experiment

The household happens to own the M3 at all three sizes: the base chip in the Air (10 GPU cores, 100 GB/s, 24 GB), the Max in the 2023 MacBook Pro (40 cores, 400 GB/s, 128 GB) and the Ultra in the office Studio (80 cores, 819 GB/s, 512 GB). Same GPU core, same memory technology, same macOS build, same MLX — only the count of things changes, and it changes in two dimensions that the vendor's table ties together: bandwidth goes 1 : 4 : 8.2 and GPU cores go 1 : 4 : 8. The physics track says prefill is the compute stage and decode the bandwidth stage. If that is true, prefill should follow the cores and decode the bandwidth, and both should scale linearly as long as nothing else intervenes. The ladder is the test.

Model (4-bit)	GB/token	air, M3 · prefill / decode	pro2023, M3 Max	office, M3 Ultra	Evidence
Qwen3.5-0.8B	0.42	1,707 / 167.6	4,017 / 416.6	6,312 / 338.2	measured 2026-09-15
Qwen3.5-2B	1.06	774 / 76.8	2,441 / 246.9	3,992 / 257.1	measured
Qwen3.5-4B	2.37	308 / 34.4	1,126 / 121.3	1,748 / 147.6	measured
Qwen3.5-9B	4.47	172 / 19.4	642 / 72.9	1,043 / 95.1	measured
Qwen3.5-27B	14.42	49 / 6.1	195 / 23.4	315 / 32.6	measured
Qwen3.5-35B-A3B	1.66	— (20 GB, not run on 24 GB)	909 / 109.9	1,276 / 89.1	measured

Base to Max: Linear

Four times the cores, four times the bandwidth. On the 9B: prefill 3.7×, decode 3.8×. On the 27B: prefill 4.0×, decode 3.8×. On the 0.8B, decode 2.5× — the fixed cost's share, which the Max's lower per-token overhead (1.12 ms against the Air's 1.45) does not fully hide at three milliseconds a token. Within the noise of a busy laptop, the Max is four Airs, and the physics track's two stages scale with the two things the vendor said they would.

Max to Ultra: Half a Second Die

Two dies, so twice the cores and twice the bandwidth on paper (2.05×). Measured on the 9B: prefill 1.62×, decode 1.30×. On the 27B: prefill 1.62×, decode 1.39×. On the mixture: prefill 1.40×, decode 0.81× — slower. The second die buys sixty per cent of its listed prefill and thirty to forty of its listed decode, and for a model with many small kernels it buys less than nothing at batch one. The previous lesson's fit says why in two numbers: the Ultra's decode kernel pulls 61% of its spec against the Max's 87%, and its fixed cost per token is 1.67 ms against 1.12. Prefill, being compute, loses less — it is the stage where eighty cores can all be busy — which is exactly the pattern the M2-versus-M3 Ultra lesson will find between generations too. None of this is a defect; it is what an interposer costs at the scale of a single token, measured in a house that owns both halves of the comparison.

What Buying Up the Ladder Actually Buys

Three things, and they are different things. Base to Max buys speed in proportion to the price of the silicon: four Airs of prefill and decode. Max to Ultra buys capacity more than speed — the 512 GB that holds a 750B-class mixture at 4 bits (about 450 GB; the store's DeepSeek-V4.1-Flash is 510 GB as shipped), which no Max can hold at all — plus a third more decode and sixty per cent more prefill on models that fit both. The Air, for its part, ran the 27B at 6.1 tokens per second inside its working set, at 88 GB/s effective, right on the ladder's fitted 89: the bottom of the ladder is slow, not broken. Your card's row for this lesson is the tier you own and the ratio you would get from the next one, read from the table, not from the spec sheet.

Code

lab_ladder.py — the M3 ladder in one table, with ratios to the base chip·python

#!/usr/bin/env python3
"""The M3 ladder, one table: base (air) / Max (pro2023) / Ultra (office), plus the
M2 Ultra (music) for the next lessons. Prefill and decode per model, and each
number's ratio to the base M3 -- bandwidth ratio is 1 : 4 : 8.2, GPU cores 1 : 4 : 8.
Standard library only. Usage: lab_ladder.py air.jsonl pro2023.jsonl office.jsonl music.jsonl"""
import json, sys

machines = []
for path in sys.argv[1:]:
    rows = {}
    for line in open(path):
        r = json.loads(line)
        if r.get("experiment") == "ladder":
            rows[r["model"].split("/")[-1]] = r
    first = next(iter(rows.values()))
    machines.append((first["alias"], first["device"], first["spec_gbps"], rows))

order = ["Qwen3.5-0.8B-4bit", "Qwen3.5-2B-4bit", "Qwen3.5-4B-4bit", "Qwen3.5-9B-4bit", "Qwen3.5-27B-4bit", "Qwen3.5-35B-A3B-4bit"]
base = machines[0][3]
print("model                 GB/tok |" + "".join(f" {m[0]:>9} {m[1].replace('Apple ',''):<9} |" for m in machines))
print("                             |" + "".join(f" {'prefill':>9} {'decode':<9} |" for _ in machines))
for name in order:
    line = f"{name:21} {base[name]['weight_bytes_per_token']/1e9:6.2f} |" if name in base else f"{name:21} {'':6} |"
    for alias, dev, spec, rows in machines:
        r = rows.get(name)
        line += f" {r['prompt_tps']:9.0f} {r['generation_tps']:<9.1f} |" if r else f" {'-':>9} {'-':<9} |"
    print(line)
print("\nratios to the base M3 (air), 9B row:")
r0 = base["Qwen3.5-9B-4bit"]
for alias, dev, spec, rows in machines:
    r = rows["Qwen3.5-9B-4bit"]
    print(f"  {alias:8} spec BW x{spec/100:4.1f}   prefill x{r['prompt_tps']/r0['prompt_tps']:4.1f}   decode x{r['generation_tps']/r0['generation_tps']:4.1f}")

# 2026-09-15, mlx 0.32.2 / mlx-lm 0.31.3, prompt 209 tokens, 200 generated, greedy, batch 1:
# model                 GB/tok |       air M3        |   pro2023 M3 Max    |    office M3 Ultra  |     music M2 Ultra  |
#                              |   prefill decode    |   prefill decode    |   prefill decode    |   prefill decode    |
# Qwen3.5-0.8B-4bit       0.42 |      1707 167.6     |      4017 416.6     |      6312 338.2     |      5396 369.4     |
# Qwen3.5-2B-4bit         1.06 |       774 76.8      |      2441 246.9     |      3992 257.1     |      3055 275.8     |
# Qwen3.5-4B-4bit         2.37 |       308 34.4      |      1126 121.3     |      1748 147.6     |      1333 153.3     |
# Qwen3.5-9B-4bit         4.47 |       172 19.4      |       642 72.9      |      1043 95.1      |       789 105.5     |
# Qwen3.5-27B-4bit       14.42 |        49 6.1       |       195 23.4      |       315 32.6      |       233 35.3      |
# Qwen3.5-35B-A3B-4bit         |         - -         |       909 109.9     |      1276 89.1      |       986 101.5     |
# ratios to the base M3, 9B row: pro2023 BW x4.0 prefill x3.7 decode x3.8; office BW x8.2 prefill x6.1 decode x4.9; music BW x8.0 prefill x4.6 decode x5.5

Exercise

Run the ladder on every Mac you can reach of one generation, or on your one Mac plus a friend's, and build the table with lab_ladder.py. Write the prefill and decode ratios for the 9B (or the largest model both hold) on your card beside the spec ratios. Then answer: which tier boundary scaled linearly, and which did not — and for the one that did not, which stage lost more?

Hint

Expect base-to-Max to be near-linear and anything that crosses an interposer to fall short, more on decode than prefill. If a boundary scales better than linear, one machine was thermally throttled or busy — repeat the slow one.