In-place Operations and the Trailing Underscore

The convention: trailing underscore = mutate

Any PyTorch op with a trailing underscore mutates its tensor in place: add_, mul_, zero_, fill_, uniform_, normal_, clamp_. The non-underscore version returns a new tensor and leaves the input unchanged.

Why care? Two reasons:

Memory. In-place avoids allocating a new tensor. For a 1B-parameter model, that matters.
Autograd safety. In-place ops can corrupt the autograd graph. PyTorch tries to detect this and raises a clear error, but you should know the rule: don't mutate a tensor that's needed for backward.

Where in-place ops are routine

optimizer.zero_grad() calls p.grad.zero_() on every parameter (or sets p.grad = None if you pass set_to_none=True, which is now the default in modern PyTorch and slightly faster).
Custom weight initialization typically uses .uniform_() / .normal_() inside a torch.no_grad() block.
EMA (exponential moving average) updates of teacher / momentum networks are textbook in-place territory.

Outside those patterns, lean on the non-mutating versions. The memory savings of an extra in-place is rarely worth the autograd risk in normal model code.

Code

Out-of-place vs in-place·python

import torch

t = torch.tensor([1.0, 2.0, 3.0])

# Out-of-place: returns new tensor, original untouched
t2 = t.add(5)
print(t)   # tensor([1., 2., 3.])
print(t2)  # tensor([6., 7., 8.])

# In-place: mutates t, returns t for chaining
t.add_(5)
print(t)   # tensor([6., 7., 8.])

# Chaining
t.mul_(2).clamp_(0, 100)
print(t)   # tensor([12., 14., 16.])

In-place inside no_grad — weight init·python

import torch
import torch.nn as nn

linear = nn.Linear(10, 4)

# Custom Xavier init — must be inside no_grad to avoid autograd-tracking
with torch.no_grad():
    bound = (6.0 / (linear.in_features + linear.out_features)) ** 0.5
    linear.weight.uniform_(-bound, bound)
    linear.bias.zero_()

print(linear.weight.std())  # roughly Xavier-shaped

Why autograd hates in-place — a tiny demo·python

import torch

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2          # y depends on x's values for the backward
y.sum().backward()  # works
print(x.grad)       # tensor([2., 4., 6.])

# Now mutate x AFTER the forward but BEFORE backward — invalid
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2
x.add_(100)         # corrupts the value autograd needs
try:
    y.sum().backward()
except RuntimeError as e:
    print(type(e).__name__, str(e)[:80])
# RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

In-place Operations and the Trailing Underscore

The convention: trailing underscore = mutate

Where in-place ops are routine

Code

External links

Exercise

Progress

Comments 0