What an Interface Gives, and What It Takes

"Every boundary you can unplug is a boundary your data has to cross."

The Gift

The previous lesson listed what an interface gives: substitution, independent pace, repair, permissionless innovation. Those are not abstractions. A PC owner in 2026 can put an NVIDIA card next to an AMD processor on an ASUS board with Micron memory, replace any one of them next year, and never ask any of those companies for permission. That is the gift, and it is large.

The Price, in Physics

An interface is a place where signals leave one piece of silicon, travel across a connector and a board, and enter another. Every such crossing costs four things:

Bandwidth. A connector has a fixed number of pins and a signalling rate that board traces can carry. PCI Express 5.0 at sixteen lanes moves about 63 GB/s in each direction; PCIe 6.0 doubles that to about 126. Inside an Apple package, the memory bus is 819 GB/s on the M3 Ultra, and the UltraFusion die-to-die link is "over 2.5TB/s". The interface is one to two orders of magnitude narrower than the thing it connects.
Latency. A memory access that stays on-package is measured in nanoseconds; a transaction that crosses PCIe is measured in microseconds, because it is serialized, packetized, and acknowledged.
Power and area. Driving a signal across a board takes far more energy than driving it across a die, and the transceivers that do it take silicon and pins. A socketed memory module is a longer, noisier path than a soldered chip — Framework, reporting AMD's engineers on the 256-bit Ryzen AI Max, wrote that "fanning out that giant 256-bit memory bus requires the LPDDR5x to be soldered" and that a socketed layout was "not possible without massively downclocking the memory".
Copies. When two processors have separate memories, data one produces and the other needs must be copied across. The copy is the interface made visible to software.

Apple's package pays none of these between CPU, GPU and memory, because there is no interface there to cross. That is the whole engineering argument for unified memory in one sentence. The whole engineering argument against it is that you also cannot cross it to reach a better part.

Where the Copy Actually Lands

Here is the part most comparisons get wrong, in both directions. Take a language model whose weights are 14 GB in 4-bit. On a discrete GPU with 32 GB of its own memory, the weights are copied across PCIe once, at load — about a quarter of a second at 63 GB/s — and every token after that is decoded from the card's own memory at the card's own bandwidth, 1,792 GB/s on an RTX 5090. The interface cost a quarter of a second, and then the card's memory, which is more than twice as fast as the Mac's, took over. Unified memory does not win that race; it loses it, and this quest says so.

Now take a model whose weights are 60 GB. They do not fit in 32 GB. Whatever does not fit lives in system memory, and the layers that live there cross PCIe every token. If 28 GB of weights sit on the wrong side of the boundary, decode cannot exceed 63 ÷ 28 ≈ 2.2 tokens per second no matter how fast the card is. On a Mac with enough unified memory the same 60 GB is read at 819 GB/s, every token, with no boundary — roughly 13 tokens per second on an M3 Ultra by the decode-ceiling arithmetic. That is the case unified memory wins, and it wins it by a factor of six or more, and it wins it precisely because the interface that made the GPU replaceable is the interface the weights have to cross.

The code block does both calculations. Note what decides the outcome: not the bandwidth of either memory, but whether the model fits on the fast side of the boundary. Capacity decides which regime you are in; bandwidth decides how fast that regime runs. Hold on to that sentence — it is most of the memory track and all of the homework track.

Code

interface_tax.py — the copy happens once if the model fits, every token if it doesn't·python

#!/usr/bin/env python3
"""Two machines, two models. Vendor bandwidths (GB/s): RTX 5090 memory 1792,
PCIe 5.0 x16 ~63 per direction, M3 Ultra unified 819. Decode ceiling =
bandwidth ÷ bytes read per token (physics, ignores everything but the bytes)."""

PCIE5_X16 = 63       # GB/s, one direction, after encoding (32 GT/s × 16 lanes × 128/130 ÷ 8)
VRAM_5090 = 1792     # GB/s, NVIDIA spec
VRAM_GB = 32
UNIFIED_M3U = 819    # GB/s, Apple spec


def discrete_gpu(weights_gb: float) -> tuple[float, float]:
    """(load seconds, decode ceiling tok/s) for a discrete card."""
    load_s = weights_gb / PCIE5_X16
    on_card = min(weights_gb, VRAM_GB)
    spilled = max(0.0, weights_gb - VRAM_GB)
    # per-token time: on-card bytes at VRAM speed + spilled bytes across PCIe
    per_token = on_card / VRAM_5090 + spilled / PCIE5_X16
    return load_s, 1 / per_token


def unified(weights_gb: float) -> tuple[float, float]:
    return 0.0, UNIFIED_M3U / weights_gb     # nothing crosses a boundary


for gb in (14, 28, 60, 120):
    d_load, d_tps = discrete_gpu(gb)
    _, u_tps = unified(gb)
    fits = "fits in VRAM " if gb <= VRAM_GB else "spills to host"
    print(f"{gb:4d} GB weights  {fits}  discrete: load {d_load:5.2f}s, ceiling {d_tps:7.1f} tok/s"
          f"   unified M3 Ultra: ceiling {u_tps:6.1f} tok/s")

Exercise

Run interface_tax.py. Then change VRAM_GB to 24 and VRAM_5090 to 1008 (an RTX 4090's GDDR6X), then VRAM_GB to 96 (an RTX PRO 6000, 1,792 GB/s), and re-run. For each card, find the smallest model size in the list where the unified Mac's ceiling overtakes the discrete card's, and write one sentence explaining why that crossover point moves with VRAM and not with the card's bandwidth.

Hint

The crossover is wherever the spill begins. Above it, the card's per-token time is dominated by the spilled bytes ÷ 63, and the card's 1,792 GB/s stops mattering; below it, the card's memory is simply faster than the Mac's. The card's bandwidth sets how much it wins by when it fits; VRAM sets where it stops fitting.