requires_grad — The Gateway to Autograd

One flag flips the whole machine on

requires_grad=True tells PyTorch "watch this tensor — record everything that happens to it so I can compute gradients later." It's the single switch that turns a calculator into a learning machine.

Three things to know:

Tensors default to requires_grad=False. You opt in.
If any input to an operation has requires_grad=True, the output also has requires_grad=True — the property propagates.
Model parameters (created via nn.Parameter inside an nn.Module) are automatically set to requires_grad=True. You almost never need to set it manually.

The grad_fn attribute on a tensor tells you which operation produced it (if any). Leaf tensors — the ones you created directly — have grad_fn=None. Intermediates have things like <AddBackward0>, <MulBackward0>, etc. We'll dive into autograd properly in the next track; for now, just know that requires_grad=True is the door, and the door is opened by either you or by nn.Parameter on your behalf.

Code

Toggling and inheriting requires_grad·python

import torch

t = torch.tensor([1.0, 2.0, 3.0])
print(t.requires_grad)  # False (default)

# Two ways to enable
t = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
t.requires_grad_(True)         # in-place version, with the underscore

# Inheritance: any op involving a tracked tensor produces a tracked tensor
y = t * 2 + 1
print(y.requires_grad)  # True
print(y.grad_fn)        # <AddBackward0>

nn.Parameter does it for you·python

import torch
import torch.nn as nn

linear = nn.Linear(10, 4)
for name, p in linear.named_parameters():
    print(name, p.requires_grad, type(p).__name__)
# weight True Parameter
# bias   True Parameter

# nn.Parameter is a subclass of Tensor that:
#   - sets requires_grad=True
#   - registers itself with the parent nn.Module so it appears in .parameters()

Freezing parameters for transfer learning·python

import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights

model = resnet18(weights=ResNet18_Weights.DEFAULT)

# Freeze the backbone
for p in model.parameters():
    p.requires_grad = False

# Replace the head with a fresh layer (which is unfrozen by default)
model.fc = nn.Linear(model.fc.in_features, 5)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable params: {trainable:,}")
# Trainable params: 2,565  (only the new head)

requires_grad — The Gateway to Autograd

One flag flips the whole machine on

Code

External links

Exercise

Progress

Comments 0