nn.Sequential and Composition Patterns

Three containers, one pattern

PyTorch's three core composition modules:

nn.Sequential(*modules) — runs modules in order, output of one feeds the next. No custom forward() needed.
nn.ModuleList([modules]) — a Python list that PyTorch can see. You write your own forward() and iterate however you want.
nn.ModuleDict({name: module}) — same idea but dict-shaped. Useful for multi-head models or branching architectures.

The "use Sequential when possible, ModuleList when you have to" rule is right for ~80% of code. Sequential reads cleanly and shows up nicely in print(model). ModuleList is for anything with branches, skip connections, or dynamic depth (e.g., a Transformer with N layers where N comes from a config).

The footgun: regular Python collections don't work

If you store layers in a plain list or dict (not the nn versions), they will not appear in model.parameters(), not move with .to(device), and not be saved by state_dict(). The bug is silent — your model trains, but only the layers PyTorch can see. Always use the nn.Module* versions.

Code

Sequential — the easy case·python

import torch.nn as nn

# Plain Sequential
mlp = nn.Sequential(
    nn.Linear(784, 256),
    nn.GELU(),
    nn.Dropout(0.1),
    nn.Linear(256, 10),
)

# Named — useful for inspection and partial freezing
from collections import OrderedDict
mlp_named = nn.Sequential(OrderedDict([
    ('fc1', nn.Linear(784, 256)),
    ('act', nn.GELU()),
    ('drop', nn.Dropout(0.1)),
    ('fc2', nn.Linear(256, 10)),
]))

# Now you can refer by name
print(mlp_named.fc1)

ModuleList — when you need control·python

import torch
import torch.nn as nn

class FlexibleMLP(nn.Module):
    def __init__(self, sizes):
        super().__init__()
        # ModuleList — a list PyTorch can see
        self.layers = nn.ModuleList([
            nn.Linear(sizes[i], sizes[i+1])
            for i in range(len(sizes) - 1)
        ])
        self.act = nn.GELU()

    def forward(self, x):
        for i, layer in enumerate(self.layers):
            x = layer(x)
            if i < len(self.layers) - 1:    # no activation on final layer
                x = self.act(x)
        return x

m = FlexibleMLP([784, 256, 128, 64, 10])
print(sum(p.numel() for p in m.parameters()))  # 234,506

The silent-bug example — plain list·python

import torch
import torch.nn as nn

class BrokenModel(nn.Module):
    def __init__(self):
        super().__init__()
        # WRONG: regular list. PyTorch can't see these.
        self.layers = [nn.Linear(10, 10) for _ in range(3)]

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

m = BrokenModel()
print(list(m.parameters()))   # [] — empty!
m.to('cpu')                    # silently moves nothing
m(torch.randn(4, 10))          # works at first call (CPU only)

# Fix: use nn.ModuleList instead

nn.Sequential and Composition Patterns

Three containers, one pattern

The footgun: regular Python collections don't work

Code

External links

Exercise

Progress

Comments 0