From Linear Regression to Deep Learning

The Same Loop, Many Times Over

You've seen the loop:

Define a parametric model (linear regression: $\overset{y}{^} = w x + b$ ).
Pick a loss function (MSE for regression).
Optimize parameters via gradient descent.
Hold out validation/test data to detect overfitting.

Now stack it. A neural network is just many linear regressions composed together with non-linear activations between them. Each layer is $y = W x + b$ followed by something like ReLU or sigmoid. The final layer is a regression (or classification) head. You optimize all the W's and b's at once via backprop (next track).

The Conceptual Continuity

Linear regression	Deep learning
2 parameters (w, b)	millions or billions of parameters
Closed-form solution	Gradient descent (no closed form)
1 line	Stacked transformations
MSE loss	MSE / cross-entropy / etc
Overfitting controlled by data + simple model	Overfitting controlled by data + dropout + weight decay + early stopping

Same chassis, more horsepower.

Track Reward

You've learned the universal learning loop: parametric model + loss + optimizer + held-out evaluation. Linear regression is the seed. Every neural network is the same idea on steroids. The next two tracks (Calculus and Backprop) give you the mechanics that make this loop work for huge models.

Code

A neural network is just stacked regressions·python

import torch
import torch.nn as nn

# A 2-layer neural network — stacked linear regressions
class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 32)        # linear regression #1
        self.fc2 = nn.Linear(32, 1)         # linear regression #2

    def forward(self, x):
        x = torch.relu(self.fc1(x))         # non-linearity in between
        return self.fc2(x)

# Two linear regressions + one ReLU = neural network.

Exercise

Compare the parameter count of (a) linear regression on 10 features, (b) a 2-layer net (10 → 32 → 1). How many parameters in each? Why does the deeper one have more?

Hint

(a) 10 weights + 1 bias = 11. (b) (10×32 + 32) + (32×1 + 1) = 352 + 33 = 385. The non-linearity earns the extra parameters by letting the model curve.