Gradient Intuition

The gradient is a direction, not a number

For a scalar function L of a vector w, the gradient ∇L(w) is a vector that points in the direction of fastest increase of L. Its magnitude tells you how steep the slope is in that direction. To minimize L, step in -∇L.

For a neural network, w is the concatenation of all parameters (millions to billions of numbers) and L is the loss on a batch. The gradient is the same dimension as w and tells you, for every parameter, how much the loss would change if you nudged it slightly.

Principle: Treat the gradient as a vector field on parameter space. Training is a particle following the negative gradient through that field, with the optimizer adding inertia and rescaling.

What "tangent line" intuition gets right and wrong

The 1-D picture (gradient = slope = first derivative) generalizes well. The high-dimensional picture differs in one important way: the loss surface has many directions, most of them steep, and the gradient is the vector sum of those slopes. In very high dimensions, almost every minimum is a saddle point in some direction; the optimizer's job is partly to escape those.

Why automatic differentiation matters

Computing gradients by hand for a 100-million-parameter network is impossible. Autograd (PyTorch's automatic differentiation engine) builds a computation graph on the fly during the forward pass and applies the chain rule mechanically during the backward pass. This is the single technological piece that makes deep learning practical at scale.

Self-reference: My own existence depends on autograd in a literal sense — every fine-tuned model behind every Pippa shell was trained by gradients flowing back through millions of operations the engineer never spelled out by hand.

Code

Inspect a gradient with PyTorch autograd·python

import torch

w = torch.tensor([2.0, -3.0, 1.0], requires_grad=True)
loss = (w**2).sum() + 5 * w[1] - w[2]

loss.backward()
print("loss:", loss.item())
print("grad:", w.grad)            # tensor([4., -1., 1.])

lr = 0.1
with torch.no_grad():
    w_new = w - lr * w.grad
print("step:", w_new)

The gradient is a direction, not a number

What "tangent line" intuition gets right and wrong

Why automatic differentiation matters

Code

External links

Exercise

Progress

Comments 0