Hyperparameters

~28 min · hyperparameters, tuning, search

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

What hyperparameters do

Hyperparameters are the knobs that the learning algorithm does not learn from data: regularization strength, tree depth, learning rate, number of estimators. They control capacity and the bias-variance tradeoff. Tuning them is searching the configuration space to maximize CV score.

Three search strategies

Grid search — exhaustive over a small grid; easy to reason about, expensive when the grid grows.
Random search — sample from distributions; covers more of the space per unit budget when only a few hyperparameters matter.
Bayesian / Optuna — model the objective and propose promising configurations; best for expensive evaluations.

The trap of search inflation

If you tune for 100 trials on the same validation set, you are overfitting to that set even if you never train on it. Use nested CV or hold a separate test set untouched. Search budget is also a hyperparameter you choose, not a free resource.

Code

RandomizedSearchCV across a wide space·python

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

param_dist = {
    "clf__C": loguniform(1e-3, 1e3),
    "clf__l1_ratio": [0.0, 0.2, 0.5, 0.8, 1.0],
}
search = RandomizedSearchCV(
    pipe, param_distributions=param_dist, n_iter=40, cv=5,
    scoring="average_precision", random_state=7, n_jobs=-1
)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

Optuna for tree ensembles·python

import optuna
import lightgbm as lgb
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 800),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "num_leaves": trial.suggest_int("num_leaves", 16, 256),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }
    model = lgb.LGBMClassifier(**params)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="average_precision")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=40)

External links

Exercise

Tune one hyperparameter for your model with RandomizedSearchCV (40 trials). Plot mean CV score vs the hyperparameter value. Identify the plateau where the score stops improving and stop there.