Flash as the Third Tier

"Three gigabytes a second is 300 times too slow for a token and fast enough for a prompt. Apple built a model around exactly that sentence and put it in a phone."

The Third Tier the Mac Already Has

Every Mac has a tier below the pool: the SSD, which the memory track measured on the Air at 2.87 GB/s reading uncached, a Studio's being faster and unmeasured here. The journey track's first lesson watched a cold model cross it: the file's pages fault in from flash at that rate, sixteen gigabytes in six seconds, once. The question this lesson asks is whether flash can be more than a loading path — whether weights can live there and be read during generation. The code block does the arithmetic for three of the quest's models. Streaming a model's active experts from flash on every token divides the SSD's bandwidth by the active bytes: about one token per second for a small mixture, a tenth of one for GLM-5.3. Loading a prompt's experts once and then decoding from the pool costs a second or two up front and runs at the pool's ceiling after. Per token, flash is about 285 times too slow against a Studio's 819 GB/s bus (34 times against the Air's own achieved 97 GB/s — the memory track's cliff); per prompt, it is a pause.

Apple's Own Answer

Apple's research group published the per-token version in 2023 — "LLM in a flash", which runs "models up to twice the size of the available DRAM" by windowing and bundling the reads, with a 4–5× and 20–25× speedup over naive loading on CPU and GPU — and then shipped the per-prompt version in a product. AFM 3 Core Advanced, the 20-billion-parameter model of the third-generation Apple Foundation Models, "activating just 1 to 4 billion parameters at a time": "the full model is stored in flash memory (NAND). Because NAND-to-DRAM bandwidth is too slow to swap weights token by token, as standard MoE models require, AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation." That is the code block's second column, written into the model's architecture: choose the experts once per prompt, page them into DRAM, decode from DRAM, and re-choose occasionally. The tier's physics chose the model's design, which is the mouse track's story told in the other direction — this time the workload was shaped to fit the layout.

What the Household Does With the Tier

The household's MLX server uses the flash tier the opposite way. Its cold cache stores a prompt's computed key-value blocks on the SSD and restores them when a matching prefix returns, "even after a server restart" — the prefill wall from the curve lesson, paid once and reloaded at 3 GB/s instead of recomputed at the compute stage's rate. The fleet track found that cache at zero bytes, because the hub's load never decodes; on a chat hub with long shared prefixes it would be the thing that makes the second question cheaper than the first. So the third tier has two honest uses on a Mac in 2026 — weights per prompt, as Apple ships in a phone, and cache blocks per prefix, as the household's server can do — and one dishonest one, weights per token, which the arithmetic rules out by a factor of three hundred. The checklist's item is to bring the first use to the Mac's runtimes, where today only the second exists.

Code

flash_tier.py — per token or per prompt: the SSD's bandwidth divided both ways·python

#!/usr/bin/env python3
"""Flash as the third tier. The Air's measured SSD read, and what streaming weights from
flash costs per token versus per prompt -- the arithmetic behind Apple's own choice to
route a NAND-resident model per prompt rather than per token."""
SSD = 2.87          # GB/s, air, F_NOCACHE 20 GiB file (T5); a Studio's SSD is faster and unmeasured here
POOL = 819.0
for name, active_gb, total_gb in [("AFM 3 Core Advanced (20B, 1-4B active)", 4 * 0.6, 20 * 0.6),
                                  ("Qwen3.8-Flash-Next (6B active)", 3.6, 108),
                                  ("GLM-5.3 (40B active)", 24.0, 452)]:
    per_token_from_flash = SSD / active_gb
    per_prompt_load = active_gb / SSD
    print(f"{name:42} experts from flash per token: {per_token_from_flash:5.1f} tok/s | load one prompt's experts once: {per_prompt_load:5.2f} s, then decode at {POOL/active_gb:4.0f} tok/s from the pool")
print("\nper token, flash is ~285x slower than a Studio's 819 GB/s bus (34x the Air's own 97); per prompt, it is a second or two -- which is why Apple's NAND-resident model 'makes routing decisions per prompt'.")
print("the household's MLX server uses the same tier the other way: KV blocks of a repeated prefix restored from SSD instead of recomputed -- zero bytes used so far, because its load never decodes.")

Exercise

Measure your SSD's uncached read rate with the memory track's cliff.py, then run flash_tier.py with it. Write on your card the per-token and per-prompt figures for the largest mixture you would want to run, and which of the tier's two honest uses — weights per prompt, cache per prefix — your own runtime can do today.

Hint

Most runtimes on a Mac in 2026 can do neither with weights and one with cache. If your per-prompt pause is under two seconds and your active bytes fit the pool, the model would be usable from flash the day a runtime routes it — which is the checklist's point.