Gradient Descent: Rolling Down the Loss Landscape

The Hill-Climbing Metaphor

Imagine the loss as a landscape — a hilly terrain where altitude = error. You want the lowest valley (minimum loss). You can't see the whole landscape, but you can feel the slope at your feet. Gradient descent says: take a step in the direction of steepest descent. Repeat until flat.

Mathematically: $θ_{t + 1} = θ_{t} - η \nabla L (θ_{t})$ . The gradient $\nabla L$ is the steepest ascent direction; we negate it to descend. $η$ (the learning rate) is your step size.

The Three Failure Modes

Learning rate too small → you crawl, take forever, possibly never converge.
Learning rate too large → you overshoot the valley, oscillate, possibly diverge to infinity.
Bad initialization → you start in a flat region (gradient ≈ 0) or a saddle point and don't move.

This is why "tuning the learning rate" is most of practical deep learning. The answer matters enormously and is rarely the default.

Variants You'll Meet

SGD — vanilla gradient descent on mini-batches. Simple, often surprisingly competitive.
Momentum — accumulate past gradients to push through small bumps.
Adam — momentum + per-parameter learning rate. Default in most modern training.
Schedules — start fast, slow down as you approach the valley. Cosine and warmup are common.

Gradient descent is the engine. The loss is the terrain. The learning rate is your step size. Most ML engineering is teaching this engine to navigate a particular terrain.

Code

Gradient descent on a 1-D bowl·python

import numpy as np

# Minimize f(w) = (w - 3)^2 — bowl with minimum at w=3
def f(w):  return (w - 3) ** 2
def df(w): return 2 * (w - 3)

w = -5.0          # start far from the minimum
lr = 0.1          # learning rate

for step in range(50):
    w -= lr * df(w)
    if step % 10 == 0:
        print(f"step {step}: w = {w:.4f}, loss = {f(w):.4f}")
print(f"final: w = {w:.4f}")    # ~3.0

Exercise

Run the gradient descent code with lr = 0.5, lr = 1.1, lr = 0.001. Predict what happens for each. Observe convergence, oscillation, divergence, or crawling.

Hint

lr=0.5: converges fast. lr=1.1: diverges (overshoots and amplifies). lr=0.001: crawls — needs many more iterations. The 'sweet spot' depends on the loss landscape.

Gradient Descent: Rolling Down the Loss Landscape

The Hill-Climbing Metaphor

The Three Failure Modes

Variants You'll Meet

Code

External links

Exercise

Progress

Comments 0