Regression: Drawing the Best Line Through Data

Learning a Line from Data

Ordinary least-squares linear regression chooses parameters so that the sum of squared vertical residuals between observed targets and predicted targets is as small as possible. It does not generally minimize perpendicular distance to the geometric line.

For a one-dimensional input, $\overset{y}{^} = w x + b$ , and the mean-squared objective is

$L (w, b) = \frac{1}{n} i = 1 \sum n (y_{i} - (w x_{i} + b))^{2} .$

The slope $w$ and intercept $b$ are learned from data. Least squares has a closed-form solution under standard conditions, although numerical code normally solves the linear system rather than forming a symbolic formula.

The Reusable Training Pattern

Choose a parameterized model.
Choose an objective that encodes the task and assumptions.
Fit parameters using an analytic solver or an optimizer.
Evaluate on data that was not used to fit those parameters.

Many neural networks use a final affine layer $W x + b$ , but that layer is not itself always a regression model. A classification head can output logits, a language model can output vocabulary scores, and architectures can use tied weights or non-linear output transformations.

Linear regression is a clean first example of parameter fitting. The model, loss, optimizer, and evaluation protocol—not “a line” alone—form the pattern that generalizes.

Code

Closed-form least squares·python

import numpy as np

# Toy dataset — y is roughly 2x + 1 plus noise
np.random.seed(42)
x = np.random.uniform(0, 10, 50)
y = 2.0 * x + 1.0 + np.random.normal(0, 1, 50)

# Closed-form linear regression (least squares)
A = np.vstack([x, np.ones_like(x)]).T   # design matrix [x, 1]
w, b = np.linalg.lstsq(A, y, rcond=None)[0]
print(f"learned: y = {w:.3f} x + {b:.3f}")     # near 2.0, 1.0

Gradient descent flavor·python

import torch

# Same problem, gradient descent flavor — the deep-learning recipe
x = torch.linspace(0, 10, 50)
y = 2.0 * x + 1.0 + torch.randn(50)

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

for step in range(200):
    pred = w * x + b
    loss = ((pred - y) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.01 * w.grad
        b -= 0.01 * b.grad
        w.grad.zero_()
        b.grad.zero_()
print(f"learned: y = {w.item():.3f} x + {b.item():.3f}")

MLX flavor — mx.grad, no requires_grad needed·python

import mlx.core as mx

# MLX flavor — functional gradient via mx.grad. No requires_grad, no .backward().
mx.random.seed(0)
x = mx.linspace(0, 10, 50)
y = 2.0 * x + 1.0 + mx.random.normal(shape=(50,))

def loss_fn(w, b, x, y):
    pred = w * x + b
    return ((pred - y) ** 2).mean()

# argnums=(0, 1) — get gradient w.r.t. the first two arguments
grad_fn = mx.grad(loss_fn, argnums=(0, 1))

w, b = mx.array(0.0), mx.array(0.0)
for step in range(200):
    gw, gb = grad_fn(w, b, x, y)
    w = w - 0.01 * gw
    b = b - 0.01 * gb
print(f"learned: y = {w.item():.3f} x + {b.item():.3f}")

Exercise

Generate 100 (x, y) points where y = -3x + 5 + Gaussian noise. Use either np.linalg.lstsq or PyTorch gradient descent to recover the slope and intercept. Compare the recovered values to the true ones (-3, 5).

Hint

If your loss isn't dropping, your learning rate is too high (oscillation) or too low (crawl). Start at 0.01 and adjust.

Learning a Line from Data

The Reusable Training Pattern

Code

External links

Exercise

Progress

Comments 0