Linear Regression Intuition

~28 min · linear, regression

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The geometry

Linear regression fits a hyperplane through the cloud of points. The coefficients are the per-unit change in the target attributable to each feature, holding the others constant. The intercept is the predicted target when all features are zero. With centered features the intercept is the mean of the target.

Why it still matters

Linear regression is the fastest baseline you can ship, the easiest to explain to a non-technical stakeholder, and the most stable under feature drift. Treat it as the floor of your performance: any fancier model must beat it on the metric you care about by a margin that justifies the maintenance cost.

Reading coefficients honestly

Coefficients are correlational, not causal. A positive coefficient on "days_since_signup" does not mean making customers older causes them to churn; it means among customers with similar other features, older ones happen to churn more. Multicollinearity scrambles individual coefficients while the prediction stays good.

Code

Plain linear regression and inspecting coefficients·python

from sklearn.linear_model import LinearRegression
import pandas as pd

lr = LinearRegression().fit(X_train, y_train)
coefs = pd.Series(lr.coef_, index=X_train.columns).sort_values(key=abs, ascending=False)
print(coefs.head(10))
print("intercept:", lr.intercept_)

Standardize before comparing coefficient magnitudes·python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

pipe = Pipeline([("scale", StandardScaler()), ("lr", LinearRegression())]).fit(X_train, y_train)
# Now coef magnitudes are comparable across features

External links

Exercise

Train a standardized linear regression on your dataset. Print the top-10 features by absolute coefficient. For each, write one sentence about whether the sign and magnitude match your prior. Investigate any surprises.