The Decode Ceiling: Bandwidth ÷ Bytes per Token

"The fastest a model can decode is the number of times per second its weights can be read. Everything else is a reason it is slower."

The Formula

If decode must read every active weight once per token, then tokens per second cannot exceed bytes per second divided by bytes per token. That is the whole formula — ceiling = bandwidth ÷ bytes per token — and it has two inputs the previous tracks have already produced: achieved memory bandwidth (the GPU track) and the model's bytes per token, which this lesson defines carefully, because getting it wrong is the commonest way the formula is misused.

Which Bytes Count

The lab script reads every tensor's size from the safetensors headers — the file is the truth, not the model card — and applies four rules. A vision tower is never read by text decode; Qwen3.5 checkpoints carry one (0.9 GB on the 9B), and counting it was this quest's first mistake. The input embedding is a table lookup — one row per token, not the whole matrix — unless the checkpoint ties it to the output head, in which case the matrix is read in full as the head. Routed experts in a mixture-of-experts layer are read k of E per token: the 35B-A3B reads 8 of 256, so its 18 GB of experts count as 0.57. Everything else — attention, linear attention, dense MLPs, norms, the head — is read once per token.

Model (4-bit)	On disk	Vision	Embedding	Experts (k/E)	Bytes per token	Evidence
Qwen3.5-0.8B	0.625 GB	0.201	0.143, tied (counted)	—	0.424 GB	physics from the headers
Qwen3.5-9B	5.95	0.912	0.572, untied (excluded)	—	4.466	physics
Qwen3.5-27B	16.05	0.921	0.715, untied (excluded)	—	14.42	physics
Qwen3.5-35B-A3B	20.39	0.893	0.286, untied	18.12 at 8/256	1.659	physics
Llama-3.2-1B	0.695	—	0.148, tied (counted)	—	0.695	physics

Prediction Against Measurement

With the achieved bandwidths from the GPU track, the ceilings and the lab's measured decode rates line up like this on office:

Model	Ceiling (638 GB/s ÷ bytes)	Measured (office)	Measured ÷ ceiling	Evidence
Qwen3.5-0.8B	1,505 tok/s	338	22%	measured 2026-09-15
Qwen3.5-2B	602	257	43%	measured
Qwen3.5-4B	269	148	55%	measured
Qwen3.5-9B	143	95	67%	measured
Qwen3.5-27B	44	32.6	74%	measured
Qwen3.5-35B-A3B	385	89	23%	measured

The ceiling is never exceeded — which is the first check on any decode number you are shown — and the fraction of it a model reaches rises with model size: 22% for the smallest, 74% for the largest dense model. The reason is a fixed cost per token that does not scale with bytes: launching each layer's kernels, sampling, the Python loop. The lab track fits it directly — on office about 1.7 milliseconds per token plus bytes ÷ 497 GB/s — and 1.7 milliseconds is most of a 0.8B token and a tenth of a 27B token. The mixture-of-experts row is the exception the experts lesson explains: its bytes are small but its kernel count is large.

So the honest form of the formula for a real machine is seconds per token = overhead + bytes per token ÷ effective bandwidth, where both constants are measured per machine and per runtime. The ceiling is the limit as the overhead goes to zero and the effective bandwidth goes to the streaming figure. Nothing you can download changes the first term; only bandwidth or fewer bytes change the second.

Code

bytes_per_token.py — the accounting, from the safetensors headers·python

#!/usr/bin/env python3
"""Bytes a token must read during decode, from the checkpoint's own headers.
Rules: skip the vision tower; skip the embedding table unless tied to the
head; count routed experts at k/E; count everything else once."""
import json, struct, sys
from pathlib import Path
from huggingface_hub import snapshot_download

repo = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/Qwen3.5-9B-4bit"
path = Path(snapshot_download(repo, local_files_only=True))
cfg = json.loads((path / "config.json").read_text()); tc = cfg.get("text_config", cfg)
E = tc.get("num_experts") or tc.get("num_local_experts") or 0
k = tc.get("num_experts_per_tok") or 0
tied = bool(cfg.get("tie_word_embeddings", tc.get("tie_word_embeddings", False)))

total = vision = embed = expert = 0
for f in sorted(path.glob("*.safetensors")):
    with f.open("rb") as fh:
        header = json.loads(fh.read(struct.unpack("<Q", fh.read(8))[0]))
    for name, meta in header.items():
        if name == "__metadata__":
            continue
        size = meta["data_offsets"][1] - meta["data_offsets"][0]
        total += size
        if "vision" in name or "visual" in name:
            vision += size
        elif "embed_tokens" in name:
            embed += size
        elif "switch_mlp" in name or ".experts." in name:
            expert += size

per_token = total - vision - (0 if tied else embed)
if E:
    per_token -= expert * (1 - k / E)
print(f"{repo}: disk {total/1e9:.3f} GB, vision {vision/1e9:.3f}, embed {embed/1e9:.3f} ({'tied' if tied else 'untied'}), "
      f"experts {expert/1e9:.3f} at {k}/{E} -> {per_token/1e9:.3f} GB per token")
for bw in (638, 391, 740, 97):
    print(f"   ceiling at {bw} GB/s: {bw*1e9/per_token:7.1f} tok/s")

Exercise

Run bytes_per_token.py on a checkpoint you have downloaded (any mlx-community 4-bit model) and write the ceiling for your Mac's achieved bandwidth on your card BEFORE running the model. Then decode 200 tokens with mlx-lm and record the measured rate. Compute measured ÷ ceiling and compare it with the office table: does your fraction sit where the model's size predicts?

Hint

Small model, low fraction; large model, high fraction. If your fraction is far above the office row for a similar size, your runtime has less per-token overhead than mlx-lm 0.31 on an Ultra — which is possible, and worth writing down with the version. If it is above 100%, your bytes per token are wrong.