~14 min · cuda, vram, pcie, nvlink, multi-gpu, physics
Level 0Spec-Sheet Skimmer
0 XP0/91 lessons0/19 achievements
0/100 XP to next level100 XP to go0% complete
"A card's memory is a cliff with a number on it: 24, 32, 96. Below it the card wins every division in the last lesson. Above it, the bus is the memory and the tax is per token."
The Wall
The last lesson's ladder had a column the bandwidth numbers cannot fix: "no fit". An RTX 4090 holds 24 GB, a 5090 32, the workstation RTX PRO 6000 96, an H100 80. A 27B at 4 bits is 14.4 GB per token plus cache and fits all of them; a 70B at 4.5 bits per weight is 39.7 GB and fits none of the consumer cards; a 235B mixture is 132 GB resident and fits nothing on a desk short of two 96 GB workstation cards or the Studio. The Mac Studio's 512 GB holds the 235B with room, at a ceiling near 66 tokens per second by the 12.4 GB its 22B active parameters read per token (six, if you wrongly divide by all 132 — the dense mistake the physics track exists to block), and a 405B dense at 4. The wall is not a performance number; it is a boolean, and the physics track's whole argument for the Mac reduces to the models for which the boolean is false on a card and true on a Studio.
Spilling: the Bus Becomes the Memory
What a runtime does when the weights exceed the card is keep the excess in host memory and stream it across PCI Express every token. Then the journey track's step 2 is taken on every token, and the ceiling is the bus's: 63 GB/s on sixteen lanes of Gen 5 gives 4.4 tokens per second for the 27B and 1.6 for the 70B — below the Air's 6.1 on the 27B, on a card whose memory is eighteen times faster than the Air's pool. This is the interface lesson's price, paid per token instead of once. Every consumer-card benchmark that shows a large model at a few tokens per second is this arithmetic, and the honest version says so: the card is not slow, it is reading through a straw.
Splitting: the Tax
The other answer is more cards. Split a 70B across two 5090s and each reads half the weights in parallel — the bandwidth adds — but every layer's activations must cross from one card to the other, and the crossing is over whatever link the cards have. The 5090's own specification page lists "NVIDIA NVLink (SLI-Ready): No"; GeForce cards talk to each other over PCIe, through the host, at the same 63 GB/s the spill used, once per layer per token, with a synchronization on each hop. Data-center parts have NVLink — 1.8 TB/s per GPU on NVLink 5 — which is why the split is a rack-scale technique that works and a desktop technique that disappoints. The tax has three lines: the link's latency times the layer count, per token; a second 575-watt power budget and a power supply the vendor lists as 1,000 W for one card; and the software to shard, which exists in the platform's twenty years and is the reason the tax is payable at all. A 235B at 4.5 bits per weight needs five 5090s, 2,875 watts of graphics power, and an all-reduce across five cards on every layer. The Studio runs it at up to 66 tokens per second by active bytes — the expert lesson says a runtime reaches a fraction of that — on 270 watts, and the fleet track is about what the household does with that.
Model, 4.5 bits/weight (the 27B row is the measured 4-bit checkpoint)
Bytes per token
RTX 5090, 32 GB
RTX PRO 6000, 96 GB
Mac Studio, 512 GB
Evidence
27B
14.4 GB
fits: 124 tok/s ceiling
fits: 124
fits: 57
derived from vendor bandwidth
70B
39.7 GB
2 cards, or 1.6 tok/s spilled
fits: 45
fits: 21
derived
235B mixture, 22B active (all experts resident)
132 GB resident; 12.4 read per token
5 cards
2 cards
fits: 66
derived
405B
228 GB
8 cards
3 cards
fits: 4
derived
Code
spill_math.py — fits, spills, or splits: the wall as arithmetic·python
#!/usr/bin/env python3
"""The VRAM wall and the multi-GPU tax, as arithmetic. A model either fits one card,
spills to the host over PCIe, or is split across cards -- and each split adds a
per-token exchange over whatever links the cards. Vendor bandwidths; bytes per
token at 4.5 bits per weight; KV at 64 KB per token (the lab's 27B figure)."""
PCIE5_X16 = 63e9 # GB/s per direction, one card's slot
CARDS = [("RTX 5090", 32, 1792), ("RTX PRO 6000", 96, 1792), ("H100 SXM", 80, 3350)]
MODELS = [("27B", 25.6e9), ("70B", 70.6e9), ("120B", 120e9), ("235B", 235e9), ("405B", 405e9)]
def bytes_per_token(params): return params * 4.5 / 8
print(f"{'model':6} {'GB/tok':>7} |" + "".join(f" {c[0]:>14}" for c in CARDS) + " | spilled over PCIe 5 x16")
for name, params in MODELS:
bpt = bytes_per_token(params)
cells = []
for cname, mem, bw in CARDS:
need = bpt / 1e9 + 4 # weights + a few GB of context and buffers
if need <= mem:
cells.append(f"{bw*1e9/bpt:10.0f} tok/s")
else:
n = -(-need // mem) # cards needed, ceiling division
cells.append(f"{int(n):3d} cards ")
print(f"{name:6} {bpt/1e9:7.1f} |" + "".join(f" {c:>14}" for c in cells) + f" | {PCIE5_X16/bpt:6.1f} tok/s")
print("\nsplit across N cards: each token still reads 1/N of the weights on each card in parallel,")
print("then exchanges activations between cards once per layer -- over PCIe (63 GB/s) on GeForce,")
print("which lists 'NVLink (SLI-Ready): No', or over NVLink (1.8 TB/s per GPU, NVLink 5) on data-center parts.")
# model GB/tok | RTX 5090 RTX PRO 6000 H100 SXM | spilled over PCIe 5 x16
# 27B 14.4 | 124 tok/s 124 tok/s 233 tok/s | 4.4 tok/s
# 70B 39.7 | 2 cards 45 tok/s 84 tok/s | 1.6 tok/s
# 120B 67.5 | 3 cards 27 tok/s 50 tok/s | 0.9 tok/s
# 235B 132.2 | 5 cards 2 cards 2 cards | 0.5 tok/s
# 405B 227.8 | 8 cards 3 cards 3 cards | 0.3 tok/s
Take the three largest models you would like to run. For each, and for one card and one Mac you could buy, write which regime it lands in — fits, spills, or splits — and the ceiling or card count from spill_math.py. Then circle the one row where the machines swap places.
Hint
The swap row is the model that fits the Mac and not the card: that row alone decides the purchase, because every other row goes to the card. If no such row exists for models you actually want, the card is the right buy and this quest says so.
Progress
Progress is local-only — sign in to sync across devices.