Backpropagation: The Chain Rule

Backprop in one paragraph

Backpropagation is the chain rule applied to a computation graph. For a network L = f₃(f₂(f₁(w₀; x))), the gradient of the loss with respect to a parameter deep in the network is the product of local Jacobians along the path from that parameter to the loss. Autograd records the graph during the forward pass; backward walks it in reverse, multiplying local Jacobians, and accumulates the gradient at each parameter.

You almost never write backprop by hand. PyTorch and JAX both implement it as a generic algorithm over a directed acyclic graph of operations, where each operation registers its own forward and backward. The reason to learn backprop deeply anyway is that it tells you why some architectures train and others do not.

The chain rule, picture form

If z = g(y) and y = f(x), then dz/dx = (dz/dy) · (dy/dx). For a deep stack z = g(f(h(...x...))), the gradient at x is the product of every local derivative along the chain. Each local derivative is small or moderate; their product can vanish or explode, which is why deep networks need normalization, residual connections, and good initialization.

Tip: Every modern stability trick — He init, BatchNorm, residual connections, gradient clipping — exists to keep that long chain product well-conditioned. They are not optional ornaments; they are why backprop works at depth.

What autograd gives you for free

Define a forward pass. Set requires_grad=True on the leaves you want gradients for. Call .backward() on a scalar loss. Read .grad on every leaf. That is the whole user interface. Inside, PyTorch built a graph, walked it backward, applied the chain rule, and accumulated the result.

Principle: Backprop is fragile because it multiplies long chains of small numbers. The architecture decisions in the next track (normalization, residual connections, careful init) all exist to keep that product alive at depth.

Code

Backprop by hand vs autograd, on a tiny graph·python

import torch

x = torch.tensor(3.0, requires_grad=True)
a = x ** 2
b = torch.exp(x)
c = a + b
L = c * 2

# By hand: dL/dx = 2 * (2x + e^x) = 2*(6 + e^3)
manual = 2 * (2 * 3 + torch.exp(torch.tensor(3.0))).item()

L.backward()
auto = x.grad.item()

print(f"manual: {manual:.4f}")
print(f"autograd: {auto:.4f}")

Vanishing gradient diagnostic·python

import torch, torch.nn as nn

class DeepSig(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(*[
            nn.Sequential(nn.Linear(64, 64), nn.Sigmoid())
            for _ in range(50)
        ])
        self.head = nn.Linear(64, 1)
    def forward(self, x):
        return self.head(self.layers(x))

m = DeepSig()
x = torch.randn(8, 64)
y = m(x).sum()
y.backward()

first = list(m.layers[0][0].parameters())[0].grad.norm().item()
last  = list(m.layers[-1][0].parameters())[0].grad.norm().item()
print(f"first-layer grad norm: {first:.6f}")
print(f"last-layer grad norm:  {last:.6f}")

Backprop in one paragraph

The chain rule, picture form

What autograd gives you for free

Code

External links

Exercise

Progress

Comments 0