no_grad, inference_mode, and detach

Three ways to say "don't track gradients here"

Most of the time you want autograd recording. Sometimes you don't — during inference, evaluation, preprocessing, metric computation, plotting. Recording costs memory (it has to keep activations around) and a small amount of CPU. Three tools for skipping it:

torch.no_grad() — context manager. No graph is built for any op inside the block. The standard wrapper around model.eval() code paths.
torch.inference_mode() — stronger version of no_grad. Disables both autograd AND view-tracking, slightly faster. The recommended choice for pure inference.
tensor.detach() — returns a new tensor that shares storage but is disconnected from the autograd graph. Used for individual tensors (logging loss values, RL target networks) rather than whole code blocks.

The boundary: `model.eval()` vs `no_grad()`

These two are related but different:

model.eval() changes module behavior — Dropout becomes a no-op, BatchNorm uses running stats instead of batch stats. It does NOT disable autograd.
torch.no_grad() disables autograd. It does NOT change module behavior.

For evaluation you almost always want both: model.eval() for the right behavior, torch.no_grad() (or inference_mode()) for speed.

Code

no_grad and inference_mode — context managers·python

import torch
import torch.nn as nn

model = nn.Linear(10, 2)
x = torch.randn(4, 10)

# no_grad: context manager
with torch.no_grad():
    y = model(x)
    print(y.requires_grad)        # False — no graph built

# inference_mode: stronger and faster
with torch.inference_mode():
    y = model(x)
    print(y.requires_grad)        # False

# As a decorator
@torch.inference_mode()
def predict(model, x):
    return model(x)

detach — disconnect a single tensor·python

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2

# Logging the value — don't drag autograd along
loss_value = y.detach().item()    # plain Python float

# A target network in RL — gradients should NOT flow into it
target = (x ** 2).detach()         # treated as a constant from now on

# detach IS view-shaped — same storage, different graph status
print(y.detach().data_ptr() == y.data_ptr())  # True

The full evaluation idiom·python

import torch
import torch.nn as nn

def evaluate(model, val_loader, criterion, device):
    model.eval()                       # behavior switch (dropout off, etc.)
    total_loss = total_correct = total_n = 0

    with torch.inference_mode():       # autograd switch (off)
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            out = model(x)
            loss = criterion(out, y)

            total_loss += loss.item() * x.size(0)
            total_correct += (out.argmax(-1) == y).sum().item()
            total_n += x.size(0)

    return total_loss / total_n, total_correct / total_n

no_grad, inference_mode, and detach

Three ways to say "don't track gradients here"

The boundary: `model.eval()` vs `no_grad()`

Code

External links

Exercise

Progress

Comments 0

Three ways to say "don't track gradients here"

The boundary: model.eval() vs no_grad()

Code

External links

Exercise

Progress

The boundary: `model.eval()` vs `no_grad()`