The Forward Pass

What forward actually means

The forward pass is the act of taking input data x through every layer in order and producing output y (logits, probabilities, embeddings — whatever the head returns). On the way, PyTorch records every operation in a computation graph so backward can compute gradients later. This is the only thing your forward() method has to do correctly.

forward() is called automatically when you do model(x). Never call model.forward(x) directly — you would skip the hooks PyTorch uses for things like normalization mode, gradient tracking, and DDP synchronization.

Tip: If you write a forward(), write the shape of every intermediate tensor next to its line. Future-you (and the next person debugging your model) will thank you.

Train mode vs eval mode

Some layers behave differently in training and evaluation: dropout (active in train, identity in eval), batch normalization (uses batch statistics in train, running statistics in eval). Switch with model.train() and model.eval(). Forgetting eval() at validation is a famous bug that produces noisy metrics; we will see it again in the training-loop track.

Stop the gradient when you do not need it

For inference and for any "frozen" backbone in transfer learning, wrap forward calls in torch.no_grad() (or torch.inference_mode() for the latest, fastest path). PyTorch will skip building the computation graph and free memory you would otherwise tie up.

Principle: Three things to remember about forward: call the model, not .forward(); switch to eval() for inference; and use inference_mode() when you do not need gradients.

Code

A complete, shape-annotated forward pass·python

import torch, torch.nn as nn

class TinyVisionMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(28 * 28, 256)
        self.act = nn.ReLU()
        self.dropout = nn.Dropout(p=0.1)
        self.fc2 = nn.Linear(256, 10)
    def forward(self, x):              # x:    [B, 1, 28, 28]
        x = self.flatten(x)            # x:    [B, 784]
        x = self.act(self.fc1(x))      # x:    [B, 256]
        x = self.dropout(x)            # x:    [B, 256]
        return self.fc2(x)             # logits: [B, 10]

model = TinyVisionMLP()
batch = torch.randn(32, 1, 28, 28)

model.train()
logits_train = model(batch)

model.eval()
with torch.inference_mode():
    logits_eval = model(batch)

print(logits_train.shape, logits_eval.shape)

What forward actually means

Train mode vs eval mode

Stop the gradient when you do not need it

Code

External links

Exercise

Progress

Comments 0