The Artificial Neuron

Three things and a name

An artificial neuron is the simplest unit of a neural network and far less mysterious than the name suggests. It does three things: take a vector of inputs x, multiply by a vector of learned weights w, add a learned bias b, and apply a non-linear activation f. That is it. The whole rest of deep learning is "stack many of these and let backprop do its job."

In maths: y = f(w · x + b). In PyTorch terms: a single output nn.Linear(in_features, 1) followed by an activation. The biological metaphor — dendrites, synapses, axon — is loose; the actual unit is just a learned linear projection plus a non-linearity.

Tip: If you can read 'matrix multiply, add bias, apply non-linearity' without flinching, you can read every neural network architecture. Every layer in this quest is a variation on those three steps.

Why the activation matters

Without a non-linear activation, every layer is a linear function of the previous one, so a stack of layers collapses to a single linear function. You cannot solve XOR with one or twenty linear layers; you can with one layer plus ReLU. This is the most important reason activations exist: they are what make depth meaningful.

Reading the shapes

Get into the habit of writing the shape next to every variable in a forward pass. x: [B, in_features], w: [out_features, in_features], y: [B, out_features]. Most bugs in your first month of PyTorch are shape mismatches, and most of those bugs disappear the moment you start writing shapes in comments.

Principle: Shapes are the contract between layers. Read the shape, predict the shape, then assert the shape. Models that train usually have shape-correct forward passes; models that crash usually do not.

Code

A single neuron, by hand and with PyTorch·python

import torch, torch.nn as nn

x = torch.randn(3, 4)                      # [B=3, in=4]

w = torch.randn(4, 1, requires_grad=True)  # [in, out]
b = torch.zeros(1, requires_grad=True)
y_manual = torch.relu(x @ w + b)           # [B, 1]

neuron = nn.Linear(in_features=4, out_features=1)
y_pytorch = torch.relu(neuron(x))          # [B, 1]

print(y_manual.shape, y_pytorch.shape)

Three things and a name

Why the activation matters

Reading the shapes

Code

External links

Exercise

Progress

Comments 0