Forward Pass: Shooting the Arrow

Computing Outputs from Current Parameters

A forward pass evaluates the model with its current parameters. For a simple multilayer perceptron, a layer may compute $h_{ℓ + 1} = σ (W_{ℓ} h_{ℓ} + b_{ℓ})$ . Other architectures add attention, convolution, normalization, recurrence, routing, or state updates.

Convert the batch into the model's expected tensors and masks.
Run each operation in dependency order to produce hidden states and outputs.
Interpret the output according to the head: logits, a regression value, an embedding, or another structured result.
During training, combine the output with targets and any auxiliary terms to compute an objective.

Frameworks usually record the operations required for automatic differentiation during this pass. Training mode can also change behavior: dropout samples a mask and BatchNorm updates statistics, while evaluation mode uses their inference behavior.

Loss Is Not the Prediction

The model may output logits while the loss function consumes those logits and target indices. A forward pass can also be used without a loss during inference or feature extraction. Parameter values do not change until an optimizer applies an update.

The forward pass computes values and records dependencies. The loss defines an objective; the backward pass computes sensitivities; the optimizer performs the update.

Code

One forward pass·python

import torch
import torch.nn as nn
import torch.nn.functional as F

# A 2-layer network — forward pass only
class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 8)
        self.fc2 = nn.Linear(8, 3)

    def forward(self, x):
        h = F.relu(self.fc1(x))      # layer 1 + ReLU
        return self.fc2(h)            # layer 2 → logits

net = TinyNet()
x = torch.randn(1, 4)                # 1 sample, 4 features
y_true = torch.tensor([2])           # true class index

logits = net(x)                       # forward pass
loss = F.cross_entropy(logits, y_true)
print(f"loss: {loss.item():.4f}")

Exercise

Define a 3-layer fully-connected network in PyTorch (input 10 → 64 → 32 → 1). Pass a random input through it. What's the shape of each intermediate layer's output?

Hint

Use nn.Linear and F.relu. Print h.shape after each layer to see the dimensionality flow: (1, 10) → (1, 64) → (1, 32) → (1, 1).

Computing Outputs from Current Parameters

Loss Is Not the Prediction

Code

Exercise

Progress

Comments 0