What Luck Did Not Fix

"The mouse was a bandwidth accident. Everything that is not bandwidth was not touched by it."

Five Things the Accident Left Alone

A story that ends at "then Apple leaned in" is a fan's story. The honest version has a last chapter listing what the follow-through has not changed, because each item is a constraint the physics track derived and a cost the household pays. Every one carries the label of the track that measured it.

The ceiling. A single pool tops out at 512 GB, and has since March 2025 — the same figure on the M5 Ultra announced 2026-08-25 (its 512 GB configuration ships late October). The follow-through added bandwidth (819 → 1,229 GB/s), accelerators and an interconnect, and did not add a byte to the largest pool one Mac can hold. The homework track asks why, as a thought experiment, and RDMA's answer — pool several Macs — is a different machine with a different price, not a bigger pool.
The batch of one. The physics track measured seven times the aggregate throughput at batch 32 on office. A household's chat window is batch 1, and the cost per token of a single user reading the weights alone is a structural property of local inference that no chip changes. The cloud amortizes; the desk does not.
Prefill. Decode was the mouse; prefill is compute-bound and was not touched by memory layout at all. Ninety seconds to first token on the 27B at 90% of a 32K window (office, measured) is the half of the workload Apple's Neural Accelerators are aimed at — and that half is the one this quest has no M5 measurement of, so the honest status is: addressed by a vendor, unverified here.
The missing fence. One pool has no isolation between GPU claimants. The GPU track told the thirteen-year story and the household's operating rules; the follow-through has not changed the arbitration layer, and the household's reading is that it will not, because the same layer serves millions of users who never hit it.
The ecosystem. CUDA is nearly twenty years of libraries, tooling and habits — the rivals track counts them — and every framework in the world was written for it first. Metal has MLX, llama.cpp, a PyTorch backend and Core ML, all real and all younger. The mouse gave Apple a hardware shape; it did not give it two decades of software.

The Position, Complete

Put the three parts of the phrase and this list side by side and you have the household's whole view of Apple silicon in the AI era. The foot was deliberate and aimed elsewhere. The mouse was real: decode wanted the shape frames wanted. The follow-through was deliberate and is admitted in full. And five constraints stand where they stood, four of them measured in this quest on the household's own Macs. The founder's summary of the machine he runs everything on is not a compliment or a complaint; it is a table with two columns, and this track has now filled both.

Code

ledger.py — the mouse story as two columns, each row pointing at the track that measured it·python

#!/usr/bin/env python3
"""What the accident fixed, what it did not, and where in this quest each
claim was measured or sourced. A table, so that neither column can be dropped."""

FIXED = [
    ("decode fits the memory layout",      "physics track: ceiling = bandwidth / bytes per token; ladder measured on 4 Macs"),
    ("large models fit at all",            "memory track: 512 GB pool; big-models track: five real checkpoints"),
    ("a first-party runtime",              "MLX, 2023-12-05; measured throughout"),
    ("bandwidth kept rising",              "vendor: 819 -> 1,229 GB/s (M5 Ultra, unmeasured here)"),
]
NOT_FIXED = [
    ("the 512 GB ceiling",                  "vendor: 512 GB in 2025-03 and 2026-08; homework track"),
    ("batch-of-one economics",             "measured: 75 -> 528 tok/s aggregate at batch 1 -> 32 (office)"),
    ("prefill is compute-bound",           "measured: 90 s TTFT at 90% of 32K on the 27B (office)"),
    ("no fence between GPU claimants",     "GPU track: 13-year lineage; operating rules"),
    ("twenty years of CUDA",               "rivals track: the library count"),
]

print("FIXED BY THE ACCIDENT (and the follow-through)")
for what, where in FIXED:
    print(f"  + {what:36} {where}")
print("\nNOT FIXED")
for what, where in NOT_FIXED:
    print(f"  - {what:36} {where}")
print(f"\n{len(FIXED)} fixed, {len(NOT_FIXED)} standing. Both columns are the position.")

Exercise

Run ledger.py. Then, for each of the five standing items, write one line on whether it affects the workload you actually run on your Mac — and how you would know. Finally, rank the five by how much they cost you personally, and compare your ranking with the household's implicit one (ceiling first, batch second).

Hint

If you never hold more than 30K tokens of context, prefill is a minor cost for you; if you run one model for one person, batch economics are your whole cost; if your model fits in 24 GB, the ceiling is someone else's problem. The list is universal; the ranking is yours.

Five Things the Accident Left Alone

The Position, Complete

Code

External links

Exercise

Progress

Comments 0