Why a Single Linear Layer Is Limited

Composition of linear maps is still linear

If layer 1 is y = W₁ x + b₁ and layer 2 is z = W₂ y + b₂, then z = W₂ W₁ x + (W₂ b₁ + b₂) — which is itself a single linear function of x. Stacking ten linear layers gives you exactly the same expressive power as one. This is why pure linear depth is wasted: you have spent ten times the parameters to compute something a single matrix could compute.

The fix is non-linearity. Insert a non-linear activation (ReLU, GELU, tanh) between every pair of linear layers and the composition is no longer linear. Now the depth means something: each layer can carve up the input space into regions and combine them in interesting ways.

Principle: Linear without non-linearity is wasted depth. A modern intuition: a layer is two operations that always travel together — a linear projection and a non-linear gate.

The XOR canonical example

XOR is the smallest function that no single linear classifier can solve. Two inputs, one output, and the four corners of the unit square: (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0. There is no straight line that puts the 1s on one side and the 0s on the other. Add one hidden layer with two ReLU units, and it solves trivially.

What this tells you about depth

Depth is not magic. It is a budget for representational complexity that activations make spendable. The representations that early layers build (edges, character n-grams) compose into mid-level concepts (textures, syllables) that compose into high-level concepts (objects, words). Take away the activations and the composition collapses.

Code

XOR: linear fails, ReLU passes·python

import torch, torch.nn as nn

X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([0., 1., 1., 0.])

class LinearOnly(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(2, 4); self.l2 = nn.Linear(4, 1)
    def forward(self, x):
        return self.l2(self.l1(x)).squeeze(-1)

class WithReLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(2, 4); self.l2 = nn.Linear(4, 1)
    def forward(self, x):
        return self.l2(torch.relu(self.l1(x))).squeeze(-1)

def train(model, steps=2000, lr=0.1):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    for _ in range(steps):
        opt.zero_grad()
        loss = ((model(X) - y) ** 2).mean()
        loss.backward(); opt.step()
    return loss.item()

print("LinearOnly final loss:", train(LinearOnly()))   # ~0.25
print("WithReLU   final loss:", train(WithReLU()))     # ~0.0

Composition of linear maps is still linear

The XOR canonical example

What this tells you about depth

Code

External links

Exercise

Progress

Comments 0