Where NVIDIA Leads: Bandwidth and Raw Compute

"For a model that fits, the card is faster, and not by a little. Say so first; the rest of the track is about the word 'fits'."

The Memory Ladder, Rival Included

The code block puts every rung on one ladder: capacity, bandwidth, the ratio between them, and the decode ceiling each implies for the same three checkpoints the physics track priced. Read the bandwidth column. A GeForce RTX 5090 carries "1792 GB/sec of total memory bandwidth" on 32 GB of GDDR7 — 2.2 times the M3 Ultra's 819. An H100 carries 3.35 TB/s of HBM3, four times. An H200, 4.8 TB/s. A B200, derived from NVIDIA's system pages, near 8. For the 27B that fits all of them, the ceilings run 57 on the Studio, 124 on the 5090, 232 on the H100, 555 on the B200. The card wins the division, and the interface lesson already conceded it: the bus is crossed once at load, and every token after that is decoded from memory that is faster than the pool. The M5 Ultra's "1.2TB/s" narrows the gap to the consumer card and does not touch the data-center parts.

Compute, Where the Gap Is Larger

Decode is bandwidth; prefill, training and image generation are compute, and there the ratio is not two or four but fifty. The lab measured office's GPU at 19.5 TFLOP/s in fp32 through MLX and 21.9 in bf16 through PyTorch. NVIDIA's H100 page lists BF16 tensor throughput of 1,979 TFLOPS "with sparsity", so 989 dense, and FP8 at 3,958 with sparsity, 1,979 dense — Hopper introduced FP8, Blackwell adds four-bit floating point, and the tensor cores that run those precisions have no counterpart in the M3 GPU. Apple's M5 generation adds "Neural Accelerators" in every GPU core and claims "up to 4.5x the peak GPU compute for AI compared to M3 Ultra", a vendor metric this quest did not measure; even taken at face value it leaves an order of magnitude to the data-center part. This is why the prefill wall from the curve lesson — 90 seconds to the first token on a 32K prompt through the 27B — is a Mac problem more than a card problem, and why the training lesson two steps ahead is short.

Links, and the Two Things Called Bandwidth

NVIDIA's third lead is between chips. NVLink 5 carries "1.8 TB/s … per GPU"; NVLink-C2C between a Grace CPU and a Hopper GPU, 900 GB/s. Apple's UltraFusion carries "over 2.5TB/s" between the M3 Ultra's two dies and "over 4.4TB/s" on the M5 Ultra — a larger number, but for a different thing: a die-to-die link inside one package, which the lab found delivers a smaller share of the memory system's bandwidth to a decode kernel, not a link between two whole chips. Between two Macs the only links are Thunderbolt 5 at 10 GB/s and Ethernet, which the big-models track prices. So the rival's ladder has three columns the Mac does not climb: memory bandwidth for a model that fits, tensor compute for anything that multiplies, and inter-chip links for anything that spans. The next lesson is the column NVIDIA does not climb.

Code

bandwidth_ladder.py — vendor bandwidths, capacities, and the ceilings they imply·python

#!/usr/bin/env python3
"""Where NVIDIA leads: the memory ladder, vendor figures only, and the decode
ceiling each rung implies for the same 4-bit checkpoints (bytes per token from
the lab's headers: 27B 14.42 GB; a 70B at 4.5 bits ≈ 39.7 GB; a 405B ≈ 228 GB)."""
rungs = [  # name, memory GB, GB/s, source
    ("Apple M3 Ultra (Mac Studio)",        512, 819,  "Apple: 819GB/s"),
    ("Apple M5 Max (MacBook Pro)",         128, 614,  "Apple: 614GB/s"),
    ("Apple M5 Ultra (Mac Studio)",        512, 1229, "Apple: '1.2TB/s'"),
    ("NVIDIA RTX 4090",                     24, 1008, "DERIVED: 21 Gbps x 384-bit / 8"),
    ("NVIDIA RTX 5090",                     32, 1792, "NVIDIA: '1792 GB/sec'"),
    ("NVIDIA RTX PRO 6000 Blackwell",       96, 1792, "NVIDIA datasheet: 1792 GB/s"),
    ("NVIDIA DGX Spark (GB10)",            128, 273,  "NVIDIA: 273 GB/s"),
    ("NVIDIA H100 SXM",                     80, 3350, "NVIDIA: 3.35TB/s"),
    ("NVIDIA H200",                        141, 4800, "NVIDIA: '4.8 terabytes per second'"),
    ("NVIDIA B200 (per GPU)",              180, 8000, "DERIVED from DGX B200 / GB200 pages"),
]
models = [("27B 4-bit", 14.42), ("70B 4-bit", 39.7), ("405B 4-bit", 228.0)]
print(f"{'device':32} {'GB':>4} {'GB/s':>5} {'GB/s per GB':>11} |" + "".join(f" {m[0]:>11}" for m in models))
for name, mem, bw, src in rungs:
    cells = []
    for mname, gb in models:
        cells.append(f"{bw/gb:11.0f}" if gb + 2 <= mem * 0.9 else f"{'no fit':>11}")
    print(f"{name:32} {mem:4d} {bw:5d} {bw/mem:11.1f} |" + " ".join(cells))
print("\nceiling = bandwidth ÷ bytes per token; 'no fit' = weights plus a little headroom exceed 90% of the memory")

# device                             GB  GB/s GB/s per GB |   27B 4-bit   70B 4-bit  405B 4-bit
# Apple M3 Ultra (Mac Studio)       512   819         1.6 |         57          21           4
# Apple M5 Max (MacBook Pro)        128   614         4.8 |         43          15      no fit
# Apple M5 Ultra (Mac Studio)       512  1229         2.4 |         85          31           5
# NVIDIA RTX 4090                    24  1008        42.0 |         70      no fit      no fit
# NVIDIA RTX 5090                    32  1792        56.0 |        124      no fit      no fit
# NVIDIA RTX PRO 6000 Blackwell      96  1792        18.7 |        124          45      no fit
# NVIDIA DGX Spark (GB10)           128   273         2.1 |         19           7      no fit
# NVIDIA H100 SXM                    80  3350        41.9 |        232          84      no fit
# NVIDIA H200                       141  4800        34.0 |        333         121      no fit
# NVIDIA B200 (per GPU)             180  8000        44.4 |        555         202      no fit

Exercise

Add your Mac to bandwidth_ladder.py with its vendor bandwidth and memory, and add one card you could buy. For the largest model you actually run, write both ceilings on your card and the ratio. Then write the second line: whether the model fits the card at all.

Hint

If it fits, the card's ceiling is the higher one by roughly the bandwidth ratio, and the honest card says so. If it does not, the ratio is meaningless and the next lesson's table is the one to fill in.