Width, Depth, and Expressive Power

Two axes that look similar but feel different

Width (more hidden units per layer) increases the number of features the network can compute in parallel at each level of abstraction. Depth (more layers) increases the number of compositional steps the network can take to build complex features from simple ones.

The universal approximation theorem says a single hidden layer with enough width can approximate any continuous function. In practice, depth is exponentially more parameter-efficient than width for problems with hierarchical structure (vision, text, code). One deep net often beats a wide-and-shallow net of the same parameter count.

Tip: If you can pick only one, depth tends to help most for natural data with compositional structure. Width helps most for diversity at a single level of abstraction. Most modern designs are deep with moderate width, not the other way around.

The gotcha: deep nets do not train themselves

Naively stacking 50 layers makes a network worse, not better, before the modern toolbox arrived. Vanishing gradients, dead neurons, and unstable activations all kill training. The whole reason we have ResNets, batch normalization, careful initialization, and skip connections is to make depth pay off.

Width has its own gotcha — VRAM grows with the square of the hidden dimension in attention and the linear of the hidden dimension in MLP layers. A 4096-wide layer is feasible; a 16384-wide layer often is not, even on a single H100.

Where this matters in practice

For research and serious production, scaling laws (more parameters trained on more tokens) reliably reduce loss along well-known curves. For a learner, the right move is: pick a published architecture and parameter count, train it on a tractable problem, then experiment with width and depth one knob at a time.

Principle: Capacity is not free. The right axis to grow first depends on whether you are starved for compositional depth or starved for parallel features.

Code

Width vs depth at constant parameter budget·python

import torch.nn as nn

def make_mlp(in_dim, widths, out_dim):
    layers, prev = [], in_dim
    for w in widths:
        layers += [nn.Linear(prev, w), nn.ReLU()]; prev = w
    layers += [nn.Linear(prev, out_dim)]
    return nn.Sequential(*layers)

wide_shallow = make_mlp(784, [512, 512],   10)
deep_narrow  = make_mlp(784, [128]*16,     10)

def n_params(m): return sum(p.numel() for p in m.parameters())
print("wide_shallow:", n_params(wide_shallow))
print("deep_narrow: ", n_params(deep_narrow))

Two axes that look similar but feel different

The gotcha: deep nets do not train themselves

Where this matters in practice

Code

External links

Exercise

Progress

Comments 0