"A benchmark you did not run is a claim with a table attached. Read it the way the lab reads its own numbers: find the stage, count the bytes, divide by the bandwidth, and see what is left over."
The Checklist
Every number in a table someone else published needs seven questions before it can go on your card. Which stage? Prefill ("pp", "prompt processing", "TTFT") scales with compute; decode ("tg", "eval rate", "tokens per second") scales with bandwidth. A table that does not say is not a table. How many bytes per token? Model parameters times bits per weight, minus what decode does not read — and "Q4" is 4.5 bits with its scales, not 4. At what context? The curve lesson showed decode falling 15–39% across a window; a rate with no context length is a rate at the top of a curve. What batch? The physics track's batch lesson: aggregate tokens per second across many streams can be seven times the single-stream number on the same bytes. Which runtime, which version? The journey track measured a 5× gap between two paths on the same bytes; a benchmark without a commit or version is a benchmark of an unknown program. Was anything speculating? The Ollama lesson: a rate above the ceiling means more than one token per pass, and the log says so if you look. Is the spec column right? The community table below lists the M3 Ultra at 800 GB/s where Apple says 819 and mixes two commits across chip generations; even a good table has rows you must correct before dividing.
The Checker
The code block turns any decode claim into two fractions: of the vendor's bandwidth ceiling, and — for chips this quest streamed — of the bandwidth a kernel actually pulled. Above 100% of the first is impossible at one token per pass. Above 100% of the second, on a machine you have measured, is either speculation or a wrong byte count. Below 15% of the first on a model larger than 4B is a busy machine, no warm-up, or the wrong stage. Everything between is a measurement of a runtime, to be compared only with other runtimes at the same bytes and context.
| Specimen | Chip | GB/token | Claimed tok/s | Of spec ceiling | Of measured | Verdict | Evidence |
|---|---|---|---|---|---|---|---|
| llama.cpp #4167, M2 Ultra 76c, 7B F16 TG | M2 Ultra | 13.48 | 41.0 | 69% | 75% | plausible — and the M2 > M3 Ultra ordering the lab found | vendor table, checked |
| llama.cpp #4167, M3 Ultra 80c, 7B F16 TG | M3 Ultra | 13.48 | 39.8 | 65% | 84% | plausible | checked |
| llama.cpp #4167, M3 Max 40c, 7B F16 TG | M3 Max | 13.48 | 25.1 | 85% | 86% | plausible — the Max's 87% fit again | checked |
| llama.cpp #4167, M3 10c, 7B Q8_0 TG | M3 | 7.16 | 12.3 | 88% | 91% | plausible — the Air's 89% fit again | checked |
| llama.cpp #4167, M3 Ultra, 7B F16 PP | M3 Ultra | 13.48 | 1,538 | 2,532% | — | not a decode number: prefill | checked |
| a neighbour quest on this site, as it read on 2026-09-15: 70B INT4 | M3 Ultra | 39.7 | 95 | 461% | 591% | impossible at one token per pass — reported to the family and repaired by its own session the same day | checked |
| this quest's ladder, Qwen3.5-27B 4-bit | M3 Ultra | 14.42 | 32.6 | 57% | 74% | plausible | measured |
| Ollama's MLX engine, 27B NVFP4 (journey track) | M3 Ultra | 14.45 | 50.6 | 89% | 114% | above the measured stream: speculation — confirmed in the log | measured |
Three Specimens Worth Learning From
The community table is the best public Apple-silicon benchmark there is, and read with the checklist it reproduces this quest: the base and Max chips near 90% of spec, the Ultras in the 60s and 70s, the M2 Ultra ahead of the M3 Ultra on decode and behind on prefill. Its rows for the newest chips carry a different commit, its M3 Ultra bandwidth is the M2's, and its Q4_0 rows sit at 45% of ceiling because a 3.8 GB token is where the fixed cost bites — all readable once you know what to look for. The neighbour quest on this site stated, in a lesson about hardware reality, that an M3 Ultra decodes a 70B at 4 bits around 95 tokens per second. Forty gigabytes a token against 819 GB/s is a ceiling near 20; the claim needed 4.6 buses. It was a wrong model — 95 is what the community table measures for a 7B on Ultra chips, filed under the wrong size — and it was reported to the family as a cross-quest contradiction rather than repaired here, which is this quest's rule for findings outside its own files; that quest's own session repaired it the same day, and its lesson now carries the correction with a dated note. Apple's own M5 claims, which the quest carries as vendor claims, sort cleanly by the first question: "up to 4x faster LLM prompt processing" is a prefill claim about the new GPU accelerators, and the "19–27% performance boost … thanks to its greater memory bandwidth" for decode is the physics track's formula in the vendor's own words — 153 over 120 GB/s is 1.275. A vendor that labels its stages is telling you which division to do; do it.