Optimizers — SGD, Adam, AdamW, and Per-Group Settings

The three optimizers you actually need

SGD (with momentum) — the classic. Cheap memory, well-understood, can match or beat Adam with proper learning-rate scheduling on vision tasks. Default for CIFAR / ImageNet replications.
Adam — adaptive learning rates per parameter via running estimates of first and second gradient moments. Great default when you don't know much about your task.
AdamW — Adam with decoupled weight decay. The actual modern standard for Transformers, fine-tuning, and most non-vision deep learning. The "Adam" people refer to in 2026 papers is almost always AdamW.

Why AdamW and not Adam

The "weight_decay" argument in Adam used to add a term to the gradient. AdamW applies weight decay after the gradient update, which is mathematically equivalent to L2 regularization on the parameters and behaves better empirically. For any modern training setup, prefer AdamW.

Per-parameter-group settings

The optimizer accepts either a flat list of parameters OR a list of parameter groups, each with their own settings. The classic use: lower learning rate on a pretrained backbone, higher on the new head. Or: no weight decay on bias / LayerNorm parameters (a Transformer convention).

Code

Three optimizers, three flavors·python

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Sequential(nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 10))

# SGD with momentum and weight decay — vision classic
opt_sgd = optim.SGD(model.parameters(), lr=1e-1, momentum=0.9, weight_decay=1e-4)

# Adam — adaptive, decoupled-weight-decay-FREE (the bad version)
opt_adam = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))

# AdamW — adaptive AND decoupled weight decay (the good version)
opt_adamw = optim.AdamW(model.parameters(), lr=1e-3, betas=(0.9, 0.999), weight_decay=0.01)

Per-group LR — pretrained backbone + fresh head·python

import torch.optim as optim
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights

backbone = resnet18(weights=ResNet18_Weights.DEFAULT)
backbone.fc = nn.Linear(backbone.fc.in_features, 5)   # new head

# Lower LR for pretrained, higher for new head
optimizer = optim.AdamW([
    {'params': [p for n, p in backbone.named_parameters() if 'fc' not in n], 'lr': 1e-5},
    {'params': backbone.fc.parameters(), 'lr': 1e-3},
], weight_decay=0.01)

for i, g in enumerate(optimizer.param_groups):
    print(f"group {i}: lr={g['lr']}, n_params={sum(p.numel() for p in g['params']):,}")

No weight decay on bias / LayerNorm — Transformer convention·python

import torch.optim as optim
import torch.nn as nn

model = nn.TransformerEncoder(nn.TransformerEncoderLayer(512, 8, batch_first=True), num_layers=6)

# Standard Transformer training trick — exclude bias and norm params from weight decay
decay, no_decay = [], []
for name, p in model.named_parameters():
    if not p.requires_grad: continue
    if p.dim() < 2 or any(k in name for k in ('bias', 'norm', 'embedding')):
        no_decay.append(p)
    else:
        decay.append(p)

optimizer = optim.AdamW([
    {'params': decay, 'weight_decay': 0.01},
    {'params': no_decay, 'weight_decay': 0.0},
], lr=1e-4)

print(f"decay: {sum(p.numel() for p in decay):,}, no_decay: {sum(p.numel() for p in no_decay):,}")

Optimizers — SGD, Adam, AdamW, and Per-Group Settings

The three optimizers you actually need

Why AdamW and not Adam

Per-parameter-group settings

Code

External links

Exercise

Progress

Comments 0