Regression: Drawing the Best Line Through Data

The Original Learning Algorithm

You have a scatter of points. You want to summarize the relationship as a line. Linear regression finds the line that minimizes total squared distance from the points to the line. That's it. That's the whole idea.

Why call it "learning"? Because you don't know the line in advance — you let the data tell you. The line's slope and intercept are the parameters learned from data. Modern deep learning is fundamentally a fancy version of this same loop: parametric model, error function, find parameters that minimize error.

The Math, Briefly

For a 1-D input $x$ and target $y$ , we want $\overset{y}{^} = w x + b$ . The parameters $w$ (slope) and $b$ (intercept) are what we learn. Loss = mean squared error:

$L (w, b) = \frac{1}{n} i = 1 \sum n (y_{i} - (w x_{i} + b))^{2}$

Find $w, b$ that minimize $L$ . For 1-D linear regression there's a closed-form solution. For neural networks there isn't — you use gradient descent.

Why It Matters

Every neural network's last layer is a linear regression with extras.
Every loss function in supervised learning generalizes the squared-distance idea.
Every "fitting" you'll do in ML is the same shape: parametric model + loss + optimizer.

Linear regression is the seed of every learning algorithm. Once you understand it, the rest of supervised learning is "more dimensions, fancier model, same loop."

Code

Closed-form least squares·python

import numpy as np

# Toy dataset — y is roughly 2x + 1 plus noise
np.random.seed(42)
x = np.random.uniform(0, 10, 50)
y = 2.0 * x + 1.0 + np.random.normal(0, 1, 50)

# Closed-form linear regression (least squares)
A = np.vstack([x, np.ones_like(x)]).T   # design matrix [x, 1]
w, b = np.linalg.lstsq(A, y, rcond=None)[0]
print(f"learned: y = {w:.3f} x + {b:.3f}")     # near 2.0, 1.0

Gradient descent flavor·python

import torch

# Same problem, gradient descent flavor — the deep-learning recipe
x = torch.linspace(0, 10, 50)
y = 2.0 * x + 1.0 + torch.randn(50)

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

for step in range(200):
    pred = w * x + b
    loss = ((pred - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.01 * w.grad
        b -= 0.01 * b.grad
        w.grad.zero_()
        b.grad.zero_()
print(f"learned: y = {w.item():.3f} x + {b.item():.3f}")

MLX flavor — mx.grad, no requires_grad needed·python

import mlx.core as mx

# MLX flavor — functional gradient via mx.grad. No requires_grad, no .backward().
mx.random.seed(0)
x = mx.linspace(0, 10, 50)
y = 2.0 * x + 1.0 + mx.random.normal(shape=(50,))

def loss_fn(w, b, x, y):
    pred = w * x + b
    return ((pred - y) ** 2).mean()

# argnums=(0, 1) — get gradient w.r.t. the first two arguments
grad_fn = mx.grad(loss_fn, argnums=(0, 1))

w, b = mx.array(0.0), mx.array(0.0)
for step in range(200):
    gw, gb = grad_fn(w, b, x, y)
    w = w - 0.01 * gw
    b = b - 0.01 * gb
print(f"learned: y = {w.item():.3f} x + {b.item():.3f}")

Exercise

Generate 100 (x, y) points where y = -3x + 5 + Gaussian noise. Use either np.linalg.lstsq or PyTorch gradient descent to recover the slope and intercept. Compare the recovered values to the true ones (-3, 5).

Hint

If your loss isn't dropping, your learning rate is too high (oscillation) or too low (crawl). Start at 0.01 and adjust.

Regression: Drawing the Best Line Through Data

The Original Learning Algorithm

The Math, Briefly

Why It Matters

Code

External links

Exercise

Progress

Comments 0