Reproducibility and Seeds

Why exact reproducibility is hard

Modern GPU code uses non-deterministic kernels for speed: parallel reductions can reorder operations, atomic adds aren't bitwise reproducible, cuDNN picks the fastest convolution algorithm at runtime. Even with seeds set, two training runs can diverge in the third decimal place after a few epochs.

What you usually want is statistical reproducibility — same final accuracy ± 0.1%, same loss curve shape — not bitwise reproducibility. Achievable with seeds, deterministic data ordering, and a few cuDNN flags.

Principle: Set seeds for: Python's random, numpy, PyTorch CPU, PyTorch CUDA, and your DataLoader's worker_init. Then accept that GPU non-determinism still costs you the third decimal place.

The seed-set ritual

Call set_seed(42) at the very top of training, before constructing the model. Use the same seed across runs you want to compare. For ablations, change one knob and keep the seed.

When you actually need bitwise determinism

For unit tests, regression tests, and certain regulated pipelines, set torch.use_deterministic_algorithms(True) and accept the slowdown (often 2x). Set CUBLAS_WORKSPACE_CONFIG=:4096:8 for matmul determinism. Save your full environment (Python version, package versions, GPU driver) — small differences across these can break determinism even with all flags set.

Tip: Reproducibility is not a binary. Most of the time, statistical reproducibility (seeds + same data + same code) is what you actually need. Bitwise determinism is a regulated-industry feature, not a default.

Code

The standard seed-set function·python

import random, os, numpy as np, torch

def set_seed(seed: int):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if torch.cuda.is_available():
        torch.cuda.manual_seed_all(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)

def worker_init_fn(worker_id):
    seed = torch.initial_seed() % 2**32
    np.random.seed(seed)
    random.seed(seed)

set_seed(42)
loader = DataLoader(ds, batch_size=32, num_workers=8,
                    worker_init_fn=worker_init_fn,
                    generator=torch.Generator().manual_seed(42))

Exercise

Run the same training script twice with the same seed. Plot the loss curves on top of each other. They should be visually indistinguishable; the small numerical differences are GPU non-determinism. Now change the seed by 1 and rerun — the curves should still look similar but no longer overlap exactly.

Why exact reproducibility is hard

The seed-set ritual

When you actually need bitwise determinism

Code

External links

Exercise

Progress

Comments 0