Mixed Precision: BF16, FP16, FP8

Training in pure FP32 is wasteful for modern hardware. Mixed precision uses lower-precision floating-point types for the bulk of the computation while keeping a few critical operations in FP32 (or higher) for numerical stability. The result: roughly 2× memory savings and 2× throughput compared to FP32, with no measurable quality loss.

Format	Bits	Range	Use case
FP32	32	±3.4 × 10³⁸	Master weights, loss scaling, optimizer state
FP16	16	±65,504	Forward/backward pass on older GPUs
BF16	16	±3.4 × 10³⁸ (same as FP32)	Modern training (Llama 3.3, DeepSeek-V3, etc.)
FP8 (E4M3, E5M2)	8	Limited	Flash Attention 3, FP8 training experiments, inference

BF16 (Brain Floating Point) is the modern default for training. It has the same exponent range as FP32 — so no overflow problems — but only 8 bits of mantissa, which is enough precision for gradients and activations. FP16 has more mantissa precision but a much smaller range, making it prone to overflow without loss scaling. BF16 made the loss-scaling dance obsolete.

Code

BF16 mixed precision in PyTorch·python

import torch
from torch.cuda.amp import autocast

# On Ampere or later (A100, H100, etc):
with autocast(dtype=torch.bfloat16):
    logits = model(input_ids)
    loss = causal_lm_loss(logits, target_ids)

# Backward pass uses BF16 automatically
loss.backward()
optimizer.step()
optimizer.zero_grad()
# No GradScaler needed for BF16 — that was an FP16 trick.

Mixed Precision: BF16, FP16, FP8

Code

External links

Exercise

Progress

Comments 0