Two Tiers Instead of One Pool

"The rival's answer to the same homework is not a bigger pool. It is two memories behind one page table: a fast one the size of today's, a slow one four times larger. The question is whether that shape belongs in a Studio."

The Shape NVIDIA Ships

GH200 puts an H100's HBM — 96 GB at a few terabytes a second — beside a Grace CPU's 480 GB of LPDDR5X at roughly a tenth of that, and joins them with a 900 GB/s coherent link so that "the CPU and GPU share a single per-process page table". Vera and Rubin do the same at the next scale: 288 GB of HBM4 at "up to 22 TB/s" per GPU beside "up to 1.5 TB" of LPDDR5X at 1.2 TB/s on the CPU. The design admits what the capacity lesson proved: a pool cannot be both wide and large, so build a small wide one and a large narrow one and let the operating system page between them. The code block prices the shape against the store's files. Anything whose active bytes live in the fast tier decodes at the fast tier's ceiling; anything that spills to the capacity tier decodes at the capacity tier's — GLM-5.3's 24 GB of active experts at 500 GB/s is 21 tokens per second, at 1.2 TB/s is 50.

The Thought Experiment on a Mac

A two-tier Studio would keep the M5 Ultra's 512 GB at 1.2 TB/s as its fast tier and hang a capacity tier — say 2 TB of LPDDR on a DGX-Spark-class 273 GB/s bus, or on modules — behind a chip-to-chip link and the page table macOS already uses to wire GPU pages. The journey track's first lesson showed the mechanism exists: mapped, touched, wired is a three-state life for a page already, and a second tier adds a fourth state, resident-but-slow. A model would load into the fast tier until it was full and the rest into the slow one, and the physics track's decode ceiling would become two ceilings, one per tier, weighted by where each token's bytes live. For a dense model that overflows the fast tier, that is the slow tier's ceiling — 11 tokens per second for GLM-5.3's experts at 273 GB/s in the code block's row — which is the capacity lesson's dead weight again with a better name. For a mixture with a routing policy that keeps the hot experts in the fast tier, it is nearly the fast tier's ceiling, and that policy is the flash lesson's per-prompt routing one level up.

Whether It Belongs in a Studio

This quest's judgment: not yet, and not for the reason it looks. The two-tier shape is a rack's answer because a rack has thousands of streams and a batch, so the slow tier serves capacity while the fast tier serves throughput. A Studio has one stream. For one stream, a model that overflows the fast tier decodes at the slow tier's rate, and the household already has a slow tier with a familiar name — the SSD, three gigabytes a second, which the next lesson treats — and a cheaper one: the cloud, which the fleet track uses for exactly the models that would live in the slow tier. What would change the judgment is the routing policy: an operating system that could keep a mixture's active experts hot per prompt, as Apple's own phone model does from flash, would make the capacity tier a place to keep a 750B model warm at the fast tier's speed. That is software homework more than silicon, and it is on the checklist.

Code

tiers.py — the rival's two-tier pools, and a two-tier Mac as arithmetic·python

#!/usr/bin/env python3
"""Two tiers instead of one pool. A fast tier at the package's edge and a capacity tier
behind a narrower link -- NVIDIA's shape at rack scale, and what it would mean at desk scale.
Vendor bandwidths; the decode column is bytes-per-token divided by the tier that holds them."""
tiers = [  # system, fast tier (GB, GB/s), capacity tier (GB, GB/s), link
    ("Mac Studio M3 Ultra today",        (512, 819),   None,          "one pool"),
    ("NVIDIA GH200",                     (96, 4000),   (480, 500),    "NVLink-C2C 900 GB/s; one page table"),
    ("NVIDIA Vera + Rubin (per GPU)",    (288, 22000), (1500, 1200),  "NVLink-C2C"),
    ("a two-tier Mac (thought experiment)", (512, 1229), (2048, 273), "a C2C-class link; not a product"),
]
print(f"{'system':36} {'fast GB':>7} {'GB/s':>6} | {'cap GB':>6} {'GB/s':>5} | 27B from fast  GLM-5.3 (24 GB active) from capacity")
for name, fast, cap, link in tiers:
    a = fast[1] / 14.42
    b = f"{cap[1]/24.0:5.0f} tok/s" if cap else "  n/a (fits fast tier: 34)"
    print(f"{name:36} {fast[0]:7d} {fast[1]:6d} | {(cap[0] if cap else 0):6d} {(cap[1] if cap else 0):5d} | {a:8.0f} tok/s      {b}    [{link}]")
print("\na capacity tier decodes at its own bandwidth: a mixture whose active experts live there runs at the slow tier's ceiling unless the routing keeps the hot experts in the fast tier.")

Exercise

Add a two-tier machine of your own design to tiers.py — your Mac's pool as the fast tier and a capacity tier of your choosing — and compute the two ceilings for the model you most want to run. Write on your card whether that model's active bytes would live in the fast tier, and if not, whether the slow tier's ceiling is one you would use.

Hint

For a dense model the answer is almost always the slow tier's ceiling and almost always too slow; for a mixture it depends on a routing policy no Mac has yet. If your design only helps mixtures, you have rediscovered why the rack shape is a rack shape.