When Linear Works and When It Breaks

Where linear shines

Linear models are excellent when the relationship between features and target is roughly additive after sensible transformations, when the dataset is small to medium, when interpretability is required, and when the team needs to ship a stable artifact that does not need a GPU.

Where linear breaks

Strong interactions — the effect of feature A depends on feature B. Add interaction terms or switch to trees.
Non-monotone relationships — y goes up then down with X. Bin the feature, use a spline, or switch to trees.
Heteroscedastic noise — variance of the residuals depends on prediction. Try log-target, weighted regression, or quantile regression.
High-cardinality categoricals — one-hot explodes. Use target encoding or trees.

The honest upgrade path

When linear breaks, do not jump straight to a deep neural net. The middle ground is gradient boosted trees. They handle interactions, non-monotonicity, missingness, and high cardinality, while still being fast to train and interpret.

Code

Add explicit interaction features for linear models·python

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge

pipe = Pipeline([
    ("poly", PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)),
    ("ridge", Ridge(alpha=1.0)),
])

When in doubt, try gradient boosting·python

import lightgbm as lgb

model = lgb.LGBMRegressor(n_estimators=500, learning_rate=0.05).fit(
    X_train, y_train, eval_set=[(X_val, y_val)], callbacks=[lgb.early_stopping(20)]
)

Where linear shines

Where linear breaks

The honest upgrade path

Code

External links

Exercise

Progress

Comments 2