Multilayer Perceptrons

The simplest neural network you should know cold

A multilayer perceptron (MLP) is just linear → non-linearity → linear → non-linearity → linear, with the last linear layer producing logits. Every architecture in this quest is a specialization or refinement of this template. CNNs replace the linears with convolutions for images; transformers add attention; recurrent nets reuse the same MLP at every timestep.

For tabular data and many simple regression/classification problems, a 2- or 3-layer MLP with 64–256 hidden units and a sensible activation (ReLU or GELU) is a reasonable starting point. It will rarely beat a tuned gradient-boosted model, but it is the right shape to learn the rest of the field on.

Shapes through an MLP

Track the shapes layer by layer. Input x: [B, in_dim], hidden h₁: [B, hidden_dim], hidden h₂: [B, hidden_dim], output y: [B, out_dim]. The weight matrix that connects two layers has shape [out, in] in PyTorch's convention; the bias has shape [out].

Tip: If your network has 'one big layer' anywhere, you might as well not have a network. The whole point of multilayer is intermediate hidden representations that downstream layers can compose.

What changes as you add depth and width

Width (more units per layer) gives you more parallel features at one level of abstraction. Depth (more layers) gives you more compositional structure across abstractions. Real networks need a mix. For most tabular problems, two or three hidden layers of moderate width (128–512) is enough; for images and text, depth and specialized architectures dominate.

Principle: Hidden representations are the entire point of an MLP. The output layer is just a thin reader on top.

Code

An MLP with explicit shape comments·python

import torch, torch.nn as nn

class MLP(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden_dim),     # [B, in_dim] -> [B, hidden_dim]
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim), # [B, hidden_dim] -> [B, hidden_dim]
            nn.ReLU(),
            nn.Linear(hidden_dim, out_dim),    # [B, hidden_dim] -> [B, out_dim]
        )
    def forward(self, x):
        return self.net(x)

model = MLP(in_dim=20, hidden_dim=128, out_dim=3)
x = torch.randn(64, 20)
logits = model(x)
print(logits.shape)  # torch.Size([64, 3])
n_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print("params:", n_params)

The simplest neural network you should know cold

Shapes through an MLP

What changes as you add depth and width

Code

External links

Exercise

Progress

Comments 0