Mixed Precision and Reproducibility

What mixed precision is

Modern GPUs run float16 (FP16) or bfloat16 (BF16) operations 2-8x faster than float32 (FP32). Mixed precision means: cast your forward pass to FP16/BF16, keep a master copy of weights in FP32, and use a GradScaler to handle the smaller dynamic range of FP16 gradients. The result is faster training and less VRAM, with usually no accuracy hit.

BF16 has the same exponent range as FP32 (so no GradScaler needed) but lower precision. FP16 is more precise but has a smaller range (need GradScaler to avoid underflow). On H100 / A100, both are fast; on Apple Silicon, BF16 is usually the right pick.

Tip: If you're training on a modern accelerator and you're not using mixed precision, you're paying 2x in time and VRAM for no good reason. torch.autocast + GradScaler is the one-line upgrade.

The standard recipe

Wrap forward and loss in torch.autocast(device_type="cuda", dtype=torch.bfloat16). For FP16, also use GradScaler: scaler.scale(loss).backward(), scaler.unscale_(opt), then clip, then scaler.step(opt), then scaler.update().

Reproducibility caveat

Mixed precision adds a small amount of non-determinism even with seeds set, because the rounding pattern of FP16/BF16 depends on operation order. Plan for statistical reproducibility (final metric ± 0.1%), not bitwise. For applications that need exact determinism, run in FP32 and accept the cost.

Principle: Mixed precision is essentially free accuracy + speed for modern hardware. The only valid reasons to skip it are: pre-Volta GPUs (no Tensor Cores), or applications where bitwise determinism is a hard requirement.

Code

Mixed-precision training in 2026 PyTorch·python

import torch
from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)

for xb, yb in train_loader:
    xb, yb = xb.to(device), yb.to(device)
    opt.zero_grad()

    with autocast(dtype=torch.bfloat16):
        logits = model(xb)
        loss = loss_fn(logits, yb)

    # FP16 needs GradScaler; BF16 doesn't (but it's harmless to use)
    scaler.scale(loss).backward()
    scaler.unscale_(opt)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(opt)
    scaler.update()

What mixed precision is

The standard recipe

Reproducibility caveat

Code

External links

Exercise

Progress

Comments 0