Optimizer — SGD, Adam, AdamW, Per-Group 설정

실제로 필요한 세 optimizer

SGD (with momentum) — 고전. 싼 메모리, 잘 이해됨, 적절한 lr scheduling 으로 vision task 에서 Adam 과 동등하거나 이김. CIFAR / ImageNet replication 의 default.
Adam — first/second gradient moment 의 running estimate 통한 per-parameter adaptive lr. task 잘 모를 때 좋은 default.
AdamW — decoupled weight decay 의 Adam. Transformer, fine-tune, 대부분 non-vision deep learning 의 실제 modern 표준. 2026 paper 의 'Adam' 은 거의 항상 AdamW.

왜 AdamW, Adam 아님

Adam 의 'weight_decay' 인자가 gradient 에 항 추가했었음. AdamW 가 weight decay 를 gradient update 후 적용 — parameter 의 L2 regularization 과 수학적 등가, 경험적으로 더 나은 행동. modern training setup 에 AdamW 선호.

Per-parameter-group 설정

optimizer 가 flat parameter list OR 각자 설정 가진 parameter group list 받음. 고전 사용: pretrained backbone 에 낮은 lr, 새 head 에 높은 lr. 또는: bias / LayerNorm parameter 에 weight decay 없음 (Transformer convention).

Code

세 optimizer, 세 맛·python

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Sequential(nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 10))

# SGD with momentum and weight decay — vision classic
opt_sgd = optim.SGD(model.parameters(), lr=1e-1, momentum=0.9, weight_decay=1e-4)

# Adam — adaptive, decoupled-weight-decay-FREE (the bad version)
opt_adam = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999))

# AdamW — adaptive AND decoupled weight decay (the good version)
opt_adamw = optim.AdamW(model.parameters(), lr=1e-3, betas=(0.9, 0.999), weight_decay=0.01)

Per-group LR — pretrained backbone + fresh head·python

import torch.optim as optim
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights

backbone = resnet18(weights=ResNet18_Weights.DEFAULT)
backbone.fc = nn.Linear(backbone.fc.in_features, 5)   # new head

# Lower LR for pretrained, higher for new head
optimizer = optim.AdamW([
    {'params': [p for n, p in backbone.named_parameters() if 'fc' not in n], 'lr': 1e-5},
    {'params': backbone.fc.parameters(), 'lr': 1e-3},
], weight_decay=0.01)

for i, g in enumerate(optimizer.param_groups):
    print(f"group {i}: lr={g['lr']}, n_params={sum(p.numel() for p in g['params']):,}")

bias / LayerNorm 에 weight decay 없음 — Transformer convention·python

import torch.optim as optim
import torch.nn as nn

model = nn.TransformerEncoder(nn.TransformerEncoderLayer(512, 8, batch_first=True), num_layers=6)

# Standard Transformer training trick — exclude bias and norm params from weight decay
decay, no_decay = [], []
for name, p in model.named_parameters():
    if not p.requires_grad: continue
    if p.dim() < 2 or any(k in name for k in ('bias', 'norm', 'embedding')):
        no_decay.append(p)
    else:
        decay.append(p)

optimizer = optim.AdamW([
    {'params': decay, 'weight_decay': 0.01},
    {'params': no_decay, 'weight_decay': 0.0},
], lr=1e-4)

print(f"decay: {sum(p.numel() for p in decay):,}, no_decay: {sum(p.numel() for p in no_decay):,}")

Optimizer — SGD, Adam, AdamW, Per-Group 설정

실제로 필요한 세 optimizer

왜 AdamW, Adam 아님

Per-parameter-group 설정

Code

External links

Exercise

Progress

댓글 0