Underfitting, Overfitting, and Regularization

Both too simple and too complex can fail

Underfitting means the model cannot capture structure even in training data: training and validation losses are both high and similar. Overfitting means it memorizes sample noise: training loss is low while validation loss is much worse. The two failures need opposite remedies.

Distinguish them with learning curves

Plot training and validation error as the amount of training data grows. Curves that meet at a poor value suggest insufficient representation or capacity; a persistent gap suggests excessive variance. Adding regularization to underfitting can make it worse, while adding features to overfitting can widen the gap. Diagnose before prescribing.

Regularization prices coefficient size

A regularized linear model minimizes prediction loss plus a coefficient penalty. This favors a less reactive solution when several models fit training data similarly. Larger α means more shrinkage; too much suppresses real signal and creates underfitting.

Ridge, Lasso, and Elastic Net

Ridge (L2) smoothly shrinks every coefficient and often stabilizes groups of correlated features. Lasso (L1) can set coefficients exactly to zero, creating a sparse model, but may arbitrarily choose among correlated features. Elastic Net combines both behaviors and is a useful high-dimensional candidate.

Scale before penalizing

The penalty operates on coefficient numbers. Features measured in dollars and percentages need different coefficient magnitudes to express the same effect, so an unscaled penalty is unfair. Put scaler and model in one pipeline and fit both inside each training fold.

Choose α with cross-validation and inspect variability

Use RidgeCV, LassoCV, or ElasticNetCV across a broad log-spaced α range. Plot mean score and fold variability. Prefer a stable plateau over a fragile peak, and expand the range if the selected value sits at an endpoint.

Confirm the selected model once

After selecting model family and α on validation, evaluate once on the frozen test set. Compare error, nonzero coefficient count, coefficient-sign stability, and serving cost. Regularization is not magic performance dust; it is a deliberate bias toward less confidence than the sample seems to permit.

Code

RidgeCV picks alpha automatically·python

from sklearn.linear_model import RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

alphas = np.logspace(-3, 3, 25)
pipe = Pipeline([("scale", StandardScaler()), ("ridge", RidgeCV(alphas=alphas))])
pipe.fit(X_train, y_train)
print("chosen alpha:", pipe.named_steps["ridge"].alpha_)

Validation curve to inspect the bias-variance trade·python

from sklearn.model_selection import validation_curve

train_scores, val_scores = validation_curve(
    Ridge(), X_train_scaled, y_train,
    param_name="alpha", param_range=alphas,
    cv=5, scoring="neg_root_mean_squared_error",
)