What Autograd Actually Does

Reverse-mode automatic differentiation, in eager Python

Autograd is PyTorch's automatic differentiation engine. It does one specific thing: when you compute a scalar (like a loss), it walks backwards through the operations that produced that scalar and accumulates the gradient of the scalar with respect to every leaf tensor that asked to track gradients.

"Reverse-mode" matters: forward-mode AD computes derivatives forwards (good when you have few inputs and many outputs); reverse-mode goes backwards (good when you have many inputs and one output — exactly the loss-vs-parameters shape of deep learning). For a model with 100M parameters and a scalar loss, reverse-mode is the only practical option.

"Dynamic graph" — what that buys you

PyTorch builds the graph on the fly, every forward pass. This means:

You can use Python control flow (if/else, while, for) inside forward(). The graph reflects the actual path taken on this particular input.
You can debug with print statements, breakpoints, even pdb. The forward pass is just Python.
The graph is thrown away after each backward call (unless you ask it to be retained). Memory stays bounded.

The cost: dynamic graphs are slightly slower to set up than static ones because there's no opportunity for ahead-of-time optimization. torch.compile() (later track) gives you the static-graph speedup without giving up the dynamic feel — but autograd itself stays dynamic.

Code

The minimal autograd loop·python

import torch

# A tracked scalar
x = torch.tensor(3.0, requires_grad=True)

# Forward — y depends on x
y = x ** 2 + 2 * x + 1   # y = 16 at x=3

# Backward — compute dy/dx
y.backward()

# Result lands on x.grad
# dy/dx = 2x + 2 = 2*3 + 2 = 8
print(x.grad)            # tensor(8.)

Dynamic graph — Python control flow inside forward·python

import torch

def f(x, branch):
    if branch == "polynomial":
        return x ** 3 - x
    else:
        return torch.sin(x) * x

# Same code path adapts based on Python data
x = torch.tensor(2.0, requires_grad=True)
y = f(x, "polynomial")
y.backward()
print(x.grad)            # 3*x^2 - 1 = 11

x.grad = None
y = f(x, "trig")
y.backward()
# d/dx (sin(x)*x) = cos(x)*x + sin(x)  ≈ -0.832 + 0.909 = 0.077
print(x.grad)

Many parameters, one scalar — the deep-learning shape·python

import torch

# A toy 'model': linear + bias
W = torch.randn(3, 2, requires_grad=True)
b = torch.randn(2, requires_grad=True)
x = torch.randn(4, 3)             # batch=4, features=3
y_true = torch.randn(4, 2)

# Forward
y_pred = x @ W + b                # (4, 2)
loss = ((y_pred - y_true) ** 2).mean()

# One backward call → gradients on every leaf
loss.backward()
print(W.grad.shape)    # torch.Size([3, 2])
print(b.grad.shape)    # torch.Size([2])

What Autograd Actually Does

Reverse-mode automatic differentiation, in eager Python

"Dynamic graph" — what that buys you

Code

External links

Exercise

Progress

Comments 0