Where the Copy Really Happens on a Discrete GPU

"Unified memory does not 'skip PCIe'. It removes the case where PCIe is on the per-token path."

Load Time: One Crossing

On a machine with a discrete GPU, a model's life begins in host memory — read from disk into RAM by the CPU — and then crosses the PCIe bus into the card's own memory, once, with a call like cudaMemcpy from a host buffer to a device buffer. The framework may pin the host pages first so the transfer can run at full bus speed. At PCIe 5.0 x16 that is about 63 GB/s, so a 14 GB model arrives in a quarter of a second and a 60 GB one, if the card could hold it, in a second. This is the only copy a well-fitted model ever makes. Everything after it happens inside the card.

Decode: Zero Crossings, If It Fits

Each generated token reads every weight once (the physics track's bytes per token) plus the KV cache accumulated so far, and writes a few kilobytes of new cache. On a discrete card all of that traffic is VRAM traffic at VRAM speed — 1,792 GB/s on an RTX 5090 — and the bus carries only the token id in and the logits or sampled token out, a few bytes. The CPU is not in the loop except to drive it. This is the case where a discrete card beats a Mac outright, and the honest version of a neighbouring quest's sentence that unified memory "skips PCIe entirely" is: when the model fits in VRAM, PCIe was never on the per-token path in the first place, and there is nothing to skip.

Decode: One Crossing per Token, Per Spilled Layer, If It Does Not

When the model does not fit, runtimes such as llama.cpp let you say how many layers go to the GPU and leave the rest on the CPU. A layer left on the host is computed by the CPU from host memory, and its activations cross the bus each token in both directions — a small transfer, but a synchronous one, and the host-side layers run at host-memory bandwidth on a processor with far less matrix throughput. Alternatively a runtime can keep the weights in host memory and stream each spilled layer's weights across the bus every token, which is the 63-GB/s-per-token path of track two. Either way the per-token cost now includes the boundary, and the card's own bandwidth stops being the number that matters. The code block walks a token through all three cases and prints where each byte went.

NVIDIA's Own "Unified Memory" Is Not This

CUDA has had a feature called unified memory — cudaMallocManaged — for a decade. It gives the CPU and GPU one address space and migrates pages between host memory and VRAM on demand. That is a convenience over the same two memories and the same bus; a page fault on the GPU still pulls the page across PCIe. NVIDIA's actual answer to Apple's layout is hardware, not an API: Grace Hopper and Grace Blackwell put the CPU and GPU memories behind the 900 GB/s NVLink-C2C link, and the DGX Spark's GB10 gives the GPU 128 GB of LPDDR5X as "coherent unified system memory". The rivals track takes those seriously. This lesson's point is narrower: the phrase "unified memory" on a spec sheet can mean a page-migration API, a coherent link between two memories, or one physical pool, and only the last one has no copy to make.

Code

token_path.py — where each byte goes, per token, in three memory layouts·python

#!/usr/bin/env python3
"""One decode step for a model with L layers of W bytes each, plus a KV cache
of K bytes, on three layouts. Prints bytes per memory tier and a time estimate.
Bandwidths GB/s: VRAM 1792 (RTX 5090), host DDR5 ~80 (dual-channel, typical),
PCIe 5.0 x16 63 per direction, unified M3 Ultra 635 (achieved, stream.py)."""

L, W = 40, 14.4e9 / 40          # a 14.4 GB model in 40 layers
K = 2.0e9                        # KV cache so far, 2 GB
ACT = 40e3                       # activations per layer crossing a boundary, ~40 KB


def layout(name, gpu_layers, streamed=False):
    on_card = gpu_layers * W
    spilled = (L - gpu_layers) * W
    t = on_card / 1792e9 + K / 1792e9                     # VRAM traffic
    crossings = 0
    if spilled:
        if streamed:                                       # weights stream across PCIe each token
            t += spilled / 63e9; crossings += (L - gpu_layers)
        else:                                              # layers computed on the host
            t += spilled / 80e9 + 2 * ACT / 63e9 * (L - gpu_layers); crossings += 2 * (L - gpu_layers)
    print(f"{name:34} card {on_card/1e9:5.1f} GB  host {spilled/1e9:5.1f} GB  crossings {crossings:3d}  ~{1/t:6.1f} tok/s")


layout("discrete, all 40 layers in VRAM", 40)
layout("discrete, 24 layers in VRAM (host)", 24)
layout("discrete, 24 layers, weights streamed", 24, streamed=True)
t = (14.4e9 + K) / 635e9
print(f"{'unified M3 Ultra (achieved 635 GB/s)':34} pool {14.4+2:5.1f} GB  host   0.0 GB  crossings   0  ~{1/t:6.1f} tok/s")
print("\nfits -> the card wins on bandwidth; spills -> the boundary is on every token.")

Exercise

Run token_path.py and then change two things: give the discrete card 96 GB (an RTX PRO 6000) and raise the model to 60 GB. Find the layer count at which the host-computed layout falls below the unified Mac's rate, and write one sentence on what a runtime's --n-gpu-layers flag is really choosing between.

Hint

It is choosing how many boundary crossings each token pays. Every layer left on the host is two crossings and a slow computation; the flag is a dial between the card's bandwidth and the bus's, and the Mac's whole pitch is that the dial does not exist below 512 GB.