When Linear Works and When It Breaks

Where linear models shine

Linear models are excellent when transformed feature effects combine roughly additively, the dataset is small to medium, interpretation matters, and the team needs a stable low-cost artifact. When performance is close, operational simplicity is a real feature.

Residuals reveal missed structure

Plot residuals against predictions and important features. Random scatter supports the additive approximation. Curves suggest nonlinear shape, fans suggest changing variance, and group-specific bands suggest a missing category or interaction. An average metric cannot expose those shapes.

When one feature's effect depends on another

Advertising may work differently by season, or discounts by customer tier. Add a small number of domain-supported interaction terms and validate them. Generating every pair creates quadratic growth and makes both estimation and explanation harder.

When the relationship bends or changes direction

Demand may fall as temperature rises and then climb again after a threshold. Bins, splines, or low-degree polynomials can express that shape. Check whether the transformation removes the residual pattern across folds; unconstrained polynomial expansion can turn dozens of inputs into tens of thousands of columns.

When error variance changes

Revenue and latency often show wider residuals at larger scales. Compare a log target, weighted regression, or quantile regression, then evaluate again in original units. Similar mean predictions can hide very different uncertainty for the cases that matter most.

When categories have too many values

Products or postal codes with thousands of values create a huge sparse one-hot matrix and difficult unseen categories. Compare leakage-safe target encoding, frequency encoding, hashing, or category-aware tree models, and verify that grouping rare values does not erase an important population.

The honest upgrade path

Repair transformations and a few justified interactions before abandoning the linear baseline. Then compare gradient-boosted trees on the identical split and metric. Include residual slices, latency, artifact size, and monitoring burden, and require a predeclared amount of durable lift. Upgrade because evidence pays for the complexity, not because the candidate is newer.

Code

Add explicit interaction features for linear models·python

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge

pipe = Pipeline([
    ("poly", PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)),
    ("ridge", Ridge(alpha=1.0)),
])

When in doubt, try gradient boosting·python

import lightgbm as lgb

model = lgb.LGBMRegressor(n_estimators=500, learning_rate=0.05).fit(
    X_train, y_train, eval_set=[(X_val, y_val)], callbacks=[lgb.early_stopping(20)]
)