~15 min · lab, regression, bandwidth, fixed-overhead, stream, measured
Level 0Spec-Sheet Skimmer
0 XP0/91 lessons0/19 achievements
0/100 XP to next level100 XP to go0% complete
"Five models, five points, one line. The slope is the bus as your runtime sees it; the intercept is everything your runtime does that is not reading."
The Fit
The physics track's decode formula has two terms: seconds per token = a fixed cost + bytes per token ÷ bandwidth. Run a ladder of dense models of the same family and quantization on one machine, put bytes per token on the x-axis and seconds per token on the y-axis, and the points fall on a line — because the fixed cost does not care about bytes and the bandwidth term is linear in them. Ordinary least squares gives the two numbers no benchmark table prints: the intercept, which is the per-token cost of launching a model's worth of kernels and sampling and looping, and the slope, whose reciprocal is the bandwidth the runtime actually pulled from the pool. The mixture-of-experts model is left out on purpose; its bytes per token are small and its kernel count is not, so it sits above the line, and the experts lesson already explained why.
Mac, 2026-09-15
Chip, spec GB/s
Fixed cost per token
Fitted bandwidth
Fitted ÷ spec
Worst residual
Evidence
air
M3, 100
1.45 ms
89 GB/s
89%
0.92 ms
measured, 5 dense models
pro2023
M3 Max, 400
1.12 ms
347 GB/s
87%
0.30 ms
measured, 5
office
M3 Ultra, 819
1.67 ms
497 GB/s
61%
0.73 ms
measured, 6 incl. Llama-1B
music
M2 Ultra, 800
1.80 ms
546 GB/s
68%
0.51 ms
measured, 5
Reading the Two Numbers
The fixed cost is 1.1 to 1.8 milliseconds a token on every Mac, and it is lowest on the single-die Max — the two Ultras pay more per token to keep two dies fed, which is the whole reason the smallest models run fastest on the M3 Max. Against a 27B token of 30 milliseconds it is noise; against a 0.8B token of 3 milliseconds it is half the time, and that is the shape of the fraction-of-ceiling column in the previous lesson. The fitted bandwidth is the more surprising number. The base M3 and the M3 Max deliver 87–89% of their spec to a decode kernel. The M3 Ultra delivers 61%; the M2 Ultra, with the same memory system a generation older, 68%. A streaming kernel that does nothing but read and add — the GPU track's stream test — gets 78% on the M3 Ultra and 92% on the M2 Ultra, so on the M3 Ultra part of the gap is the memory system and part is the decode kernel's own inefficiency at feeding eighty cores from two dies, while on the M2 Ultra — streaming 92%, in the single-die band — the gap is almost all the kernel's. The lab cannot separate those two from user space; it reports both numbers and the gap between them.
What This Number Is Good For
It predicts. With a and BW for your Mac, the decode rate of any dense model you have not downloaded is 1 ÷ (a + bytes ÷ BW), and on these four machines the worst miss over the ladder is under a millisecond a token. It is a floor, not a ceiling, on the bus: a better kernel can pull more (the journey track's PyTorch matvec pulled 605 GB/s on office against this fit's 497), and nothing can pull more than the spec. And it exposes speculation: a rate that implies more than the streamed bandwidth is more than one token per pass, as the Ollama lesson found. The card gets both numbers and the date, because a runtime release can move either.
Code
lab_fit.py — seconds per token against bytes per token, fitted per Mac·python
#!/usr/bin/env python3
"""Bandwidth from decode speed. Over the dense models of one machine's ladder,
fit seconds-per-token = a + bytes-per-token / BW. The slope's reciprocal is the
bandwidth your runtime actually pulled; the intercept is its fixed cost per token.
Standard library only. Usage: lab_fit.py ladder-office.jsonl [more.jsonl ...]"""
import json, sys
def dense_rows(path):
for line in open(path):
r = json.loads(line)
if r.get("experiment") == "ladder" and r.get("experts", 0) == 0: # dense only: experts break the straight line
yield r
def fit(xs, ys): # ordinary least squares, y = a + b x
n = len(xs); mx = sum(xs) / n; my = sum(ys) / n
b = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
return my - b * mx, b
for path in sys.argv[1:]:
rs = list(dense_rows(path))
xs = [r["weight_bytes_per_token"] for r in rs] # bytes read per token
ys = [1.0 / r["generation_tps"] for r in rs] # seconds per token
a, b = fit(xs, ys)
bw, spec = 1.0 / b, rs[0].get("spec_gbps")
worst = max(abs(y - (a + b * x)) for x, y in zip(xs, ys))
print(f"{rs[0]['alias']:8} {rs[0]['device']:16} {len(rs)} dense models: "
f"fixed {a*1e3:.2f} ms/token + bytes / {bw/1e9:.0f} GB/s"
+ (f" ({bw/1e9/spec*100:.0f}% of spec {spec:.0f})" if spec else "")
+ f" worst residual {worst*1e3:.2f} ms")
for r, x, y in zip(rs, xs, ys):
print(f" {r['model'].split('/')[-1]:30} {x/1e9:6.2f} GB measured {1/y:7.1f} tok/s fit {1/(a+b*x):7.1f}")
# office Apple M3 Ultra 6 dense models: fixed 1.67 ms/token + bytes / 497 GB/s (61% of spec 819) worst residual 0.73 ms
# pro2023 Apple M3 Max 5 dense models: fixed 1.12 ms/token + bytes / 347 GB/s (87% of spec 400) worst residual 0.30 ms
# music Apple M2 Ultra 5 dense models: fixed 1.80 ms/token + bytes / 546 GB/s (68% of spec 800) worst residual 0.51 ms
# air Apple M3 5 dense models: fixed 1.45 ms/token + bytes / 89 GB/s (89% of spec 100) worst residual 0.92 ms
Run the ladder on your Mac with at least four dense models of one family and one quantization, then lab_fit.py on the result file. Write the fixed cost and the fitted bandwidth on your card with the date, and predict — before downloading — the decode rate of one model you do not have yet. Then download it, run it, and record the miss.
Hint
A miss under a millisecond a token is the lab's experience. A larger miss usually means the new model is a different architecture (a pure transformer against hybrids, or a mixture), which changes the fixed cost — that is a finding about the model, not a failure of the line.
Progress
Progress is local-only — sign in to sync across devices.