Weight Decay and Early Stopping

Weight decay in one sentence

Weight decay nudges parameters toward zero at every step, which prevents any one weight from dominating and acts as a smooth implicit regularizer. In AdamW, it's a separate decoupled step (subtract λw from the parameter directly). In SGD, it's mathematically identical to adding an L2 penalty to the loss.

Default values: weight_decay=0.01 for transformers, weight_decay=1e-4 for vision CNNs trained from scratch. Tune in factors of 10. Apply only to weight matrices, not to biases or LayerNorm parameters.

Tip: If you copy weight_decay=0.01 from a transformer recipe and apply it to a small MLP, training will look fine but will generalize worse. Tune for your task — don't cargo-cult the defaults from another domain.

Early stopping done right

Track validation loss every epoch. Save the model whenever val loss reaches a new low. If val loss doesn't improve for patience epochs, stop training. The saved best checkpoint is what you ship.

Common values: patience=10 for short training, patience=50 for long training. Add a min_delta so noisy 0.0001 improvements don't reset the patience counter.

The interaction

Weight decay and early stopping are roughly equivalent regularizers in practice — they both keep the model from memorizing noise. Use both, but don't expect them to compose linearly. If you have strong augmentation, you can run with less of both.

Principle: Always save the best-validation checkpoint. Always have early stopping configured even if you don't expect to use it. The cost is two lines of code; the cost of forgetting them is shipping a worse model than the one your training run already produced.

Code

Early stopping with patience·python

class EarlyStopper:
    def __init__(self, patience=10, min_delta=1e-4):
        self.patience = patience
        self.min_delta = min_delta
        self.best = float("inf")
        self.bad = 0
    def step(self, val_loss):
        if val_loss < self.best - self.min_delta:
            self.best = val_loss
            self.bad = 0
            return False         # don't stop
        else:
            self.bad += 1
            return self.bad >= self.patience

stopper = EarlyStopper(patience=10)
for epoch in range(1000):
    train_one_epoch(model, train_loader)
    val_loss = evaluate(model, val_loader)
    if val_loss < stopper.best:
        torch.save(model.state_dict(), "best.pt")
    if stopper.step(val_loss):
        print(f"Stopping at epoch {epoch}, best val_loss={stopper.best:.4f}")
        break

Weight decay in one sentence

Early stopping done right

The interaction

Code

External links

Exercise

Progress

Comments 0