Underfitting, Overfitting, and Regularization

~30 min · regularization, ridge, lasso

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The two failure modes

Underfitting: the model cannot capture the structure even on training data. Training loss is high; validation loss is similar. Fix by adding capacity, more features, or better representations. Overfitting: the model memorizes training data. Training loss is tiny; validation loss is much worse. Fix by reducing capacity, regularizing, or getting more data.

Ridge, Lasso, Elastic Net

Ridge (L2) shrinks coefficients toward zero without forcing any to be exactly zero. Lasso (L1) shrinks and performs feature selection by zeroing out small coefficients. Elastic Net is a weighted combination, often the safest default for high-dimensional tabular data. The tuning knob is α (alpha): bigger means more shrinkage.

How to tune

Use RidgeCV, LassoCV, or ElasticNetCV with cross-validation to pick alpha automatically. Plot the validation curve so you can see whether you are at the bottom or near a cliff. Always run regularization inside a pipeline that scales features first.

Code

RidgeCV picks alpha automatically·python

from sklearn.linear_model import RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

alphas = np.logspace(-3, 3, 25)
pipe = Pipeline([("scale", StandardScaler()), ("ridge", RidgeCV(alphas=alphas))])
pipe.fit(X_train, y_train)
print("chosen alpha:", pipe.named_steps["ridge"].alpha_)

Validation curve to inspect the bias-variance trade·python

from sklearn.model_selection import validation_curve

train_scores, val_scores = validation_curve(
    Ridge(), X_train_scaled, y_train,
    param_name="alpha", param_range=alphas,
    cv=5, scoring="neg_root_mean_squared_error",
)

External links

Exercise

Train Ridge, Lasso, and ElasticNet on the same problem with cross-validated alpha. Compare RMSE on the test set and count of non-zero coefficients. Report which one you would ship and why.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousRegression Metrics: MAE, RMSE, R² Next →When Linear Works and When It Breaks

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.