Optimizers: SGD to AdamW

The optimizer is the recipe for using gradients

SGD takes w := w - lr * grad. Modern optimizers add momentum (running averages of past gradients), per-parameter scaling (running averages of past squared gradients), and weight decay. The differences are what make some optimizers train faster, some generalize better, and some be the right default for transformers vs CNNs.

SGD with momentum — classic baseline. Often best for CNNs trained from scratch on ImageNet-style data. Tune lr, momentum (0.9), weight decay carefully.
Adam — adaptive per-parameter learning rates via running mean and variance of gradients. Trains fast, sometimes generalizes worse than SGD on vision.
AdamW — Adam with decoupled weight decay (the right way to do weight decay in adaptive optimizers). Default for transformers and most modern training.
Lion — sign-based optimizer, less memory than Adam, comparable accuracy on some workloads.
Adafactor — memory-efficient (no per-parameter variance state), used at very large scales (e.g., T5).

Tip: In 2026, the boring choice is AdamW with lr=3e-4 (or scaled to your batch size), weight_decay=0.01, betas=(0.9, 0.95) for transformers. Start there and only deviate when you have a reason.

Why decoupled weight decay matters

"Weight decay" was originally an L2 regularizer added to the loss: L + 0.5 * λ * ||w||². In SGD, this is mathematically identical to subtracting λw from the gradient before stepping. In Adam, it is not — the adaptive learning rate scales the L2 term in a way that makes it less effective. AdamW decouples it: subtract λw directly during the step, separate from the adaptive scaling. This small fix makes a measurable difference on most tasks.

Principle: If you ever see Adam used with weight decay in code from before 2018, suspect the original paper used L2 regularization (broken with Adam) rather than decoupled weight decay (the correct AdamW form).

Code

AdamW with sane defaults for a transformer·python

import torch
from torch import optim

optimizer = optim.AdamW(
    model.parameters(),
    lr=3e-4,
    betas=(0.9, 0.95),
    weight_decay=0.01,
)

def split_params(model):
    decay, no_decay = [], []
    for name, p in model.named_parameters():
        if not p.requires_grad: continue
        if p.ndim == 1 or name.endswith(".bias"):
            no_decay.append(p)
        else:
            decay.append(p)
    return [
        {"params": decay,    "weight_decay": 0.01},
        {"params": no_decay, "weight_decay": 0.0},
    ]

optimizer = optim.AdamW(split_params(model), lr=3e-4, betas=(0.9, 0.95))

The optimizer is the recipe for using gradients

Why decoupled weight decay matters

Code

External links

Exercise

Progress

Comments 0