Optimizers — Adam, SGD, AdamW, Schedules

~13 min · optimizer, adam, sgd, schedule

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

How weights actually move

Optimizers determine how weights update given gradients. Choice and hyperparameters significantly affect training speed and final quality.

Optimizer	Best for	Notes
SGD + momentum	Large-scale vision, when you tune LR	Often achieves best final accuracy with proper tuning
Adam	General default, NLP, early experiments	Robust, fast convergence, slightly worse peak than tuned SGD
AdamW	Transformers, large models	Decoupled weight decay — cleaner regularization
RMSprop	RNNs, non-stationary objectives	Older default before Adam dominated

Learning rate schedules almost always help. The classic pattern: warmup for a few epochs (linear from 0 to base_lr), then cosine decay back toward 0. CosineDecay with optional warmup_steps is built into Keras 3.

Code

Optimizer 4종 + clipping·python

from tensorflow import keras

sgd = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True)
adam = keras.optimizers.Adam(learning_rate=1e-3, beta_1=0.9, beta_2=0.999)
adamw = keras.optimizers.AdamW(learning_rate=1e-3, weight_decay=0.004)
rmsprop = keras.optimizers.RMSprop(learning_rate=1e-3, rho=0.9)

# Gradient clipping (works with any optimizer)
adam_clip_norm = keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)
adam_clip_val = keras.optimizers.Adam(learning_rate=1e-3, clipvalue=0.5)

Cosine decay schedule·python

from tensorflow import keras

# Smooth decrease from initial_lr to alpha * initial_lr
cosine = keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=1e-3,
    decay_steps=10_000,
    alpha=0.0,
)

# Use schedule with any optimizer
optimizer = keras.optimizers.Adam(learning_rate=cosine)

# Exponential decay
exp = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=1000,
    decay_rate=0.96,
    staircase=True,
)

Progress

Progress is local-only — sign in to sync across devices.

← Previousmodel.compile() — Wiring Optimizer, Loss, Metrics Next →Loss Functions and Metrics — The Right Choice

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.