Training Is NVIDIA's Ground

"Decode reads the weights. Training reads them, writes them, keeps two extra copies for the optimizer, and multiplies everything by six per token. The pool was shaped for the first sentence."

A Different Budget

Every lesson so far priced inference: bytes read per token over bandwidth. Training is a different budget in both memory and compute. In memory, full fine-tuning with the usual optimizer keeps, per parameter, the bf16 weight, its gradient, two fp32 optimizer moments and an fp32 master copy — about sixteen bytes — before a single activation is stored; a 7B needs 112 GB before its batch, which is why the 128 GB laptop cannot and the 512 GB Studio can, and why the big-models track's checkpoints are inference-only at those sizes. In compute, one training token costs about six floating-point operations per parameter: a forward pass and a backward pass twice its size. A billion tokens through a 7B is 42 exaFLOP. That number does not care how wide the memory bus is; it cares how many multiplications a second the machine can do.

The Fifty-to-One

The lab measured office's GPU at 19.5 TFLOP/s in fp32 and 21.9 in bf16. NVIDIA's H100 page lists BF16 tensor throughput of 1,979 TFLOPS with sparsity — 989 dense — and FP8 at 1,979 dense. The code block does the division: the billion-token fine-tune is 598 hours on office at perfect utilization and 12 on one H100 in bf16, 6 in FP8, at the same perfect utilization neither reaches. Published large runs report model-FLOPs utilization well below peak; a laptop GPU through PyTorch, the journey track showed, reaches far less of its own. Scale the training up — a real pretraining run is trillions of tokens on tens of billions of parameters — and the Mac's number is not weeks but years, against a rack's days. The M5 generation's accelerators change the Mac's figure by Apple's claimed 4.5× at most; they do not change its order of magnitude against a data-center part, and the interconnect lesson said why the Mac cannot rack up: Thunderbolt at 10 GB/s against NVLink at 1,800.

Where Apple Trains, and What a Mac Trains

Apple's own account of its foundation models: they are trained on "Apple's AXLearn framework", which "allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs". The company that designed the pool trains on other people's silicon, and says so. What a Mac trains is the smaller thing that the mlx quest teaches: fine-tuning with low-rank adapters, where the trainable parameters are a few per cent of the model and the sixteen-byte budget applies only to those; small models from scratch; embeddings; the household's own classifiers. That is real work and the household does it. It is not the ground the rival owns, and this quest does not pretend the pool competes there. The rivalry is settled by task: inference on models that fit the pool and not the card goes to the Mac; training at any scale that matters goes to NVIDIA; the next lesson is where NVIDIA starts building pools of its own.

Code

training_math.py — bytes per parameter, FLOPs per token, hours per machine·python

#!/usr/bin/env python3
"""Training is a different budget: bytes and FLOPs. Full fine-tuning in bf16 with Adam
needs about 16 bytes per parameter before activations; a training step costs ~6 FLOPs per
parameter per token. Compute rates are vendor peak figures (dense, no sparsity) and the
lab's measured MLX matmul on office; real utilization is a fraction of either."""
params = 7e9
tokens = 1e9
bytes_full = params * 16                           # bf16 weights 2 + grads 2 + Adam m,v fp32 8 + master fp32 4
flops = 6 * params * tokens
rates = [  # name, TFLOP/s, note
    ("M3 Ultra, MLX fp32 matmul (measured)",      19.5,   "this quest, CPU track"),
    ("M3 Ultra, PyTorch MPS bf16 matmul (measured)", 21.9, "this quest, journey track"),
    ("H100 SXM, BF16 tensor core, dense",          989.5,  "NVIDIA: 1,979 TFLOPS with sparsity, halved"),
    ("H100 SXM, FP8 tensor core, dense",           1979,   "NVIDIA: 3,958 TFLOPS with sparsity, halved"),
]
print(f"full fine-tune of a {params/1e9:.0f}B: {bytes_full/1e9:.0f} GB of optimizer state and weights before activations")
print(f"{tokens/1e9:.0f}B training tokens: {flops/1e18:.0f} EFLOP\n")
for name, tflops, note in rates:
    secs = flops / (tflops * 1e12)
    print(f"{name:52} {tflops:7.1f} TFLOP/s -> {secs/3600:8.1f} h at 100% utilization   [{note}]")
print("\nutilization is never 100%: published large runs report model-FLOPs utilization well below peak, and a laptop GPU through PyTorch far below that.")

# full fine-tune of a 7B: 112 GB of optimizer state and weights before activations
# 1B training tokens: 42 EFLOP
# M3 Ultra, MLX fp32 matmul (measured)                    19.5 TFLOP/s ->    598.3 h at 100% utilization
# M3 Ultra, PyTorch MPS bf16 matmul (measured)            21.9 TFLOP/s ->    532.7 h at 100% utilization
# H100 SXM, BF16 tensor core, dense                      989.5 TFLOP/s ->     11.8 h at 100% utilization
# H100 SXM, FP8 tensor core, dense                      1979.0 TFLOP/s ->      5.9 h at 100% utilization

Exercise

Run training_math.py with the parameter count of a model you would like to fine-tune and the number of tokens you would use. Write the memory for a full fine-tune and for a low-rank adapter at 2% of parameters, and the hours on your Mac at its measured matmul rate. Decide, on the card, which of the two your Mac can actually do.

Hint

If the full fine-tune's 16 bytes per parameter exceeds your working set, the answer is the adapter; if the hours run to weeks, the answer is the adapter or a rented card. Both are honest outcomes — the pool's shape favours inference, and this quest says so in the rival's track.