What It Means to Learn

Learning is gradient descent on a scalar

For a neural network, learning means: pick a scalar loss that measures how wrong the model is on a batch, compute the gradient of that loss with respect to every parameter, and nudge each parameter in the opposite direction of its gradient. Repeat for many batches. That is it. Every other concept in this track — optimizers, schedules, regularization — is a refinement of this loop.

The reason the loop works is calculus. A gradient tells you the direction in parameter space that increases loss the fastest; stepping in the opposite direction decreases loss. Take small enough steps and you crawl downhill toward a local minimum.

Principle: Learning, in deep learning, is just downhill walking on a (very high-dimensional) loss surface. Everything else is implementation detail.

Why this looks magical and isn't

The magic is that the loss surface, even for a 70-billion-parameter model, has a structure that gradient descent can exploit. Empirically, big neural nets find low loss in regions that also generalize. Why this works is still an open research question; that it works is well established and is what makes the field interesting.

What you actually do as an engineer

Pick a loss that matches your task (classification → cross-entropy, regression → MSE, ranking → triplet). Pick an optimizer (AdamW is a fine default in 2026). Pick a learning rate (start with 1e-4 to 3e-3 and tune). Run your training loop. Watch the loss curves. Iterate.

Tip: Your first deep-learning intuition project should be: implement gradient descent on a 2-D quadratic loss surface by hand, plot the trajectory, and watch how the learning rate changes the path. Twenty minutes of that and the rest of the track makes physical sense.

Code

Gradient descent on a 2-D bowl·python

import numpy as np
import matplotlib.pyplot as plt

def loss(w):  return 0.5 * (w[0]**2 + 4 * w[1]**2)
def grad(w):  return np.array([w[0], 4 * w[1]])

def gd(start, lr, steps):
    path = [np.array(start, dtype=float)]
    for _ in range(steps):
        w = path[-1]
        path.append(w - lr * grad(w))
    return np.array(path)

slow   = gd([2.0, 2.0], lr=0.1,  steps=50)   # gentle, smooth
fast   = gd([2.0, 2.0], lr=0.4,  steps=50)   # zig-zags
toobig = gd([2.0, 2.0], lr=0.55, steps=50)   # diverges

for name, path in [("lr=0.1", slow), ("lr=0.4", fast), ("lr=0.55", toobig)]:
    plt.plot(path[:, 0], path[:, 1], label=name)
plt.legend(); plt.show()

Learning is gradient descent on a scalar

Why this looks magical and isn't

What you actually do as an engineer

Code

External links

Exercise

Progress

Comments 0