From Linear Regression to Deep Learning

A Shared Optimization Loop

Define a parameterized model.
Choose an objective.
Use an optimizer or solver to fit parameters.
Use validation for selection and held-out data for final evaluation.

A feed-forward neural network commonly composes affine transformations $W x + b$ with nonlinear activations. Without the nonlinear steps, a stack of affine layers collapses to one affine transformation. Other neural architectures also include attention, normalization, convolutions, recurrence, routing, or state updates, so “stacked regressions” is only a first bridge.

Continuity and Difference

Linear least squares	Deep learning
Often convex with an analytic or direct numerical solution	Usually nonconvex and trained iteratively
A small, fixed feature map	Learns many intermediate representations
MSE under common assumptions	Task-specific objectives such as cross-entropy, contrastive loss, or policy objectives
Simple capacity control	Data scale, architecture, augmentation, regularization, optimization, and early stopping all interact

GPT-style language modeling is usually described as self-supervised next-token prediction: labels are derived from the text itself. It uses the same optimization pattern, but not externally labeled supervised regression.

Track Reward

You now have the reusable chassis: model, objective, optimization, selection, and evaluation. Linear regression makes the chassis visible; calculus and backpropagation explain how large differentiable models update millions or billions of parameters together.

Code

Affine layers plus a nonlinearity·python

import torch
import torch.nn as nn

# A two-layer feed-forward network: affine maps plus a nonlinearity
class TinyNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 32)
        self.fc2 = nn.Linear(32, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        return self.fc2(x)

# Without ReLU, the two affine layers would collapse to one affine map.

Exercise

Compare the parameter count of (a) linear regression on 10 features, (b) a 2-layer net (10 → 32 → 1). How many parameters in each? Why does the deeper one have more?

Hint

(a) 10 weights + 1 bias = 11. (b) (10×32 + 32) + (32×1 + 1) = 352 + 33 = 385. The non-linearity earns the extra parameters by letting the model curve.

A Shared Optimization Loop

Continuity and Difference

Track Reward

Code

External links

Exercise

Progress

Comments 0