Backpropagation: The Chain Rule Doing Bookkeeping

The Question Backprop Answers

You did a forward pass. You got a loss. Now you need to know: "For each weight in this network — possibly billions of them — how would the loss change if I nudged that weight a tiny bit?" Those numbers are the gradients. With them, gradient descent can update every weight to reduce loss.

Naively, you'd compute each gradient separately by perturbing one weight at a time. That's $O (N)$ extra forward passes for $N$ weights. With billions of weights, this would take forever.

Backprop's Trick

Backpropagation computes ALL gradients in essentially one extra pass through the network — backwards. The math behind this is the chain rule (you saw it in the Calculus track). The engineering is: cache intermediate values during the forward pass, then walk backward, multiplying local Jacobians as you go.

For each weight $w$ in layer $ℓ$ :

$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial h _{L}} \cdot \frac{\partial h _{L}}{\partial h _{L - 1}} \cdot \dots \cdot \frac{\partial h _{ℓ + 1}}{\partial h _{ℓ}} \cdot \frac{\partial h _{ℓ}}{\partial w}$

That's a chain of derivatives. Backprop computes each once, reuses them across all weights.

The Blame-Game Metaphor

Think of backprop as blame propagation. The loss says "we're off the bullseye by this much." Backprop walks backward through the network, asking each layer: "how much did you contribute to the miss?" Each layer answers in proportion to its weight × the gradient flowing in from above. Then it passes the (chained) blame to the layer behind it.

By the end, every weight knows its share of the error. Adjust each weight against its share, and the next forward pass gets closer to the bullseye.

Backprop = chain rule + dynamic programming. It computes gradients for every parameter in a single backward pass by reusing intermediate derivatives instead of recomputing them.

Code

One backward call, all gradients·python

import torch

# Manual two-layer net to see backprop in action
x = torch.randn(1, 4)
W1 = torch.randn(4, 8, requires_grad=True)
W2 = torch.randn(8, 1, requires_grad=True)

# Forward
h = torch.relu(x @ W1)
y_hat = h @ W2
loss = (y_hat ** 2).sum()        # arbitrary loss for demo

# Backward — chain rule, automated
loss.backward()
print(W1.grad.shape)             # torch.Size([4, 8])
print(W2.grad.shape)             # torch.Size([8, 1])
# Both gradients computed in ONE backward call.

Backpropagation: The Chain Rule Doing Bookkeeping

The Question Backprop Answers

Backprop's Trick

The Blame-Game Metaphor

Code

External links

Exercise

Progress

Comments 0