Write the Prediction Before the Number

"A number you did not predict is a number you cannot be surprised by. The lab's first rule is to make surprise possible."

The Protocol

Every measured number in this quest came from one small program, silicon_lab.py, written for the quest and run on four of the household's Macs on one day — the base M3 in the Air, the M3 Max in the 2023 MacBook Pro, the M3 Ultra in the office Studio, the M2 Ultra in the music Studio. It does five things in a fixed order, and the order is the protocol. Stamp: before anything runs, record the machine by role alias, the chip as MLX names it, the memory and the recommended working set, the macOS build, the MLX and mlx-lm versions, and the UTC time — a number without these is a rumour. Count the bytes: open the checkpoint's safetensors headers and add up what a decode step actually reads, by the physics track's rules: skip the vision tower, skip the embedding table unless it is tied to the output head, count routed experts at k of E. The config file says what the model is; the headers say what is on disk. Predict: divide the chip's spec bandwidth by those bytes and print the ceiling before loading the model. Warm up: generate eight tokens and throw them away, so the first-forward page-touching from the journey track is not in the timing. Measure: a fixed prompt of about 209 tokens, 200 greedy tokens out, batch one, and record prefill rate, time to first token, decode rate and peak memory, with the prediction beside them.

Why the Prediction Comes First

The first office ladder was wrong, and the prediction is how it was caught. That run counted every byte on disk as bytes read per token: 0.63 GB for the 0.8B instead of 0.42, 5.95 for the 9B instead of 4.47, 20.39 for the mixture instead of 1.66. The decode rates it measured were fine; the effective bandwidth it derived from them was not — 1,815 GB/s for the mixture, on a chip whose bus carries 819. A number above the ceiling is not a fast machine, it is a wrong count, and the only reason that line stood out on a screen of plausible-looking rates was that the ceiling had been printed next to it. Run 1 was discarded; the byte rules above were written; the physics track's expert lesson carries the confession. Without a prediction, the wrong number would have been the result.

What a Measurement Record Contains

One line per run, JSON, appended to a file the machine keeps. The lab's records carry the stamp, the byte accounting (total, vision, embedding, experts, tied, per-token), the quantization as the config states it, the prompt and generation token counts, the four rates, the peak memory, and the prediction with the effective bandwidth beside it. Every table in this quest is built from those lines by scripts you can run on the same files, and the measurements companion the quest keeps is the human-readable index of them. The Mac card you have been filling is the same idea by hand: alias, date, build, versions, model, bits, bytes per token, prediction, measurement — and the gap. The table below predicts from the vendor's 819 GB/s; the physics track's version of the same rows predicted from the 638 GB/s a kernel actually streamed on office, which is why its fractions read higher (22% and 74% at the ends) — same measurements, two honest denominators, and the card should say which one it used.

Office, 2026-09-15, mlx 0.32.2	Bytes per token	Predicted ceiling (spec 819 GB/s)	Measured decode	Fraction of spec ceiling	Evidence
Qwen3.5-0.8B-4bit	0.42 GB	1,932 tok/s	338.2	18%	measured
Qwen3.5-9B-4bit	4.47 GB	183	95.1	52%	measured
Qwen3.5-27B-4bit	14.42 GB	56.8	32.6	57%	measured
Qwen3.5-35B-A3B-4bit (run 1, wrong count)	20.39 GB	40.2	89.0	221% — impossible	measured, discarded
Qwen3.5-35B-A3B-4bit (corrected)	1.66 GB	494	89.1	18%	measured

Code

silicon_lab.py (excerpt) — stamp, count the bytes from the headers, predict, then run·python

def stamp(alias: str) -> dict:
    """Who, what, when: a measurement without this is a rumour."""
    info = mx.device_info()
    return {
        "alias": alias,
        "device": info.get("device_name"),
        "memory_gb": round(info.get("memory_size", 0) / 1e9, 1),
        "recommended_working_set_gb": round(info.get("max_recommended_working_set_size", 0) / 1e9, 1),
        "macos": macos_build(),
        "mlx": mx.__version__,
        "mlx_lm": mlx_lm.__version__,
        "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }


def weight_bytes(path: Path) -> dict:
    """Bytes a token must read during decode — from the safetensors headers, not the config.
    Not every byte on disk is read per token: a vision tower never is; the input embedding is a
    one-row lookup unless tied to the output head; a mixture reads k of E routed experts."""
    cfg = json.loads((path / "config.json").read_text())
    text_cfg = cfg.get("text_config", cfg)
    n_experts = text_cfg.get("num_experts") or text_cfg.get("num_local_experts") or 0
    k = text_cfg.get("num_experts_per_tok") or 0
    tied = bool(cfg.get("tie_word_embeddings", text_cfg.get("tie_word_embeddings", False)))
    total = vision = embed = expert = 0
    for f in sorted(path.glob("*.safetensors")):
        with f.open("rb") as fh:
            n = struct.unpack("<Q", fh.read(8))[0]
            header = json.loads(fh.read(n))
        for name, meta in header.items():
            if name == "__metadata__":
                continue
            a, b = meta["data_offsets"]
            size = b - a
            total += size
            if "vision" in name or "visual" in name:
                vision += size
            elif "embed_tokens" in name:
                embed += size
            elif "switch_mlp" in name or ".experts." in name:
                expert += size
    touched = total - vision - (0 if tied else embed)
    if n_experts:
        touched -= expert * (1 - k / n_experts)
    return {"weight_bytes_total": total, "weight_bytes_vision": vision, "weight_bytes_embed": embed,
            "weight_bytes_experts": expert, "tied_embeddings": tied, "experts": n_experts,
            "experts_per_token": k, "weight_bytes_per_token": int(touched)}


def ladder(args) -> None:
    base = stamp(args.alias)
    for repo in args.models:
        path = snapshot_dir(repo)
        wb = weight_bytes(path)
        predicted = args.spec_gbps * 1e9 / wb["weight_bytes_per_token"]          # THE PREDICTION, before load()
        print(f"prediction: {args.spec_gbps} GB/s / {wb['weight_bytes_per_token']/1e9:.2f} GB = {predicted:.1f} tok/s ceiling")
        model, tokenizer = load(str(path))
        prompt_ids = build_prompt(tokenizer, args.prompt_tokens, QUESTION)
        run_once(model, tokenizer, prompt_ids, 8)                                  # warm the kernels; discard
        rec = {**base, "experiment": "ladder", "model": repo, **wb}
        rec.update(run_once(model, tokenizer, prompt_ids, args.max_tokens))        # prefill, TTFT, decode, peak
        rec["predicted_decode_tps"] = round(predicted, 1)
        rec["effective_gbps"] = round(rec["generation_tps"] * wb["weight_bytes_per_token"] / 1e9, 1)
        with Path(args.out).open("a") as fh:
            fh.write(json.dumps(rec) + "\n")                                        # one JSON line per run; the file is the record

# office, run 1 (every byte on disk counted per token), 2026-09-15:
#   Qwen3.5-35B-A3B-4bit  20.39 GB/tok  predicted 40.2  measured 88.99  -> effective 1814.6 GB/s on an 819 GB/s bus
# corrected: 1.66 GB/tok, predicted 493.7, measured 89.1, effective 147.9 GB/s

Exercise

Before you run anything: for one 4-bit model in your cache, open its safetensors headers, apply the three byte rules, and write your Mac's ceiling on the card — spec bandwidth ÷ bytes per token. Only then run the lab's ladder on that single model and write the measured decode beside it. If the fraction is above 100%, find the counting error before you do anything else.

Hint

The usual counting errors, in order of frequency: the vision tower (a fifth of a small VLM's bytes), the embedding table on an untied model, and a mixture's experts counted in full. If the fraction is below 15% on a model larger than 4B, the run was not warmed up or the machine was busy — repeat it.