Accelerators and Devices

What you might run on in 2026

NVIDIA H100 / H200 / B100 — the workhorse of cloud training. BF16, FP8, big VRAM (80GB+).
NVIDIA A100 — still common, slightly older but plentiful in cloud.
NVIDIA RTX 4090 / 5090 — single-GPU desktop training; 24-32GB VRAM.
Apple Silicon (M3 Ultra, M4 Max) — unified memory architecture; 64-512GB shared between CPU and GPU. Fast for inference and small-to-medium training via MPS or MLX.
Google TPU (v5e, v5p) — Google Cloud only; great for very large model training; needs JAX or PyTorch/XLA.
AMD MI300X — emerging alternative to H100; ROCm support in PyTorch is improving.

Tip: Most learning and most production inference happens on a single accelerator. Multi-GPU and multi-node training is a step you should put off until you genuinely need it; the engineering tax is real.

Memory is usually the binding constraint

Modern accelerators have a lot of FLOPS. The constraint is fitting the model + activations + gradients + optimizer state in VRAM. Mixed precision halves the model + activation memory. Gradient checkpointing trades compute for memory. ZeRO and FSDP shard optimizer state across GPUs.

Apple Silicon is real

For local inference and modest training, Apple Silicon is competitive. The unified memory means a 192GB Mac Studio can hold model weights that would need multiple GPUs on x86. MLX is Apple's native framework; PyTorch's MPS backend is the cross-platform path. Both are first-class in 2026.

Principle: Pick your accelerator based on memory, not FLOPS. A 24GB consumer GPU often beats a 16GB datacenter GPU for LLM fine-tuning because the model fits.

Code

Universal device picker·python

import torch

def best_device():
    if torch.cuda.is_available():
        return torch.device("cuda")
    if torch.backends.mps.is_available():
        return torch.device("mps")
    return torch.device("cpu")

device = best_device()
print("device:", device)

if device.type == "cuda":
    print("model:", torch.cuda.get_device_name(0))
    print("VRAM (GB):", torch.cuda.get_device_properties(0).total_memory / 1e9)

What you might run on in 2026

Memory is usually the binding constraint

Apple Silicon is real

Code

External links

Exercise

Progress

Comments 0