Why Images Are Different

The structure that pixels actually have

An image isn't a flat vector. It's a 2-D grid where nearby pixels are correlated, the same object can appear at different positions, and useful features are local before they're global. A naively-flattened image fed into an MLP loses all of this. CNNs were invented to exploit that structure directly.

Three properties matter: locality (a feature lives in a local window of pixels), translation invariance (a cat is a cat wherever it appears in the frame), and compositionality (low-level features compose into mid-level features compose into objects).

Tip: If a model wastes parameters re-learning that translation doesn't change the label, it has fewer parameters left for the actual task. Architectures that bake translation invariance in (CNNs) outperform architectures that have to learn it (early MLPs on images).

The cost of treating images as flat vectors

An MLP on 224×224 RGB images has 3 * 224 * 224 = 150,528 input features. The first hidden layer has roughly that many parameters per hidden unit. A 256-wide first layer alone uses 38M parameters — more than a small CNN — and learns a unique weight for every (input pixel, hidden unit) pair. That's a lot of parameters memorizing translation that a single 3×3 convolution captures with 9 weights.

Where this leads

The next lessons in this track build the toolkit: convolutions (locality + translation), pooling (downsampling), the classic CNN lineage (AlexNet → VGG → ResNet → ConvNeXt), then ViT (transformers eat vision too), then sequence models, then attention, then transformer blocks. The pattern: each architecture encodes assumptions about the data; pick the architecture whose assumptions match.

Principle: Architecture is opinion. Every layer choice is an assumption about what structure the data has. Knowing the assumptions is what lets you debug a stuck model.

Code

MLP vs CNN parameter counts on a tiny image task·python

import torch.nn as nn

# MLP on 32x32 RGB
mlp = nn.Sequential(
    nn.Flatten(),
    nn.Linear(3 * 32 * 32, 128), nn.ReLU(),
    nn.Linear(128, 10),
)
print("MLP params:", sum(p.numel() for p in mlp.parameters()))

# Small CNN on 32x32 RGB
cnn = nn.Sequential(
    nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.AdaptiveAvgPool2d(1), nn.Flatten(),
    nn.Linear(64, 10),
)
print("CNN params:", sum(p.numel() for p in cnn.parameters()))
# CNN typically uses fewer parameters AND generalizes better.

The structure that pixels actually have

The cost of treating images as flat vectors

Where this leads

Code

External links

Exercise

Progress

Comments 0