Hyperparameters

Hyperparameters control how learning happens

Regularization strength, tree depth, learning rate, and estimator count are not ordinary fitted coefficients. They control capacity, optimization, and the bias-variance tradeoff. Tuning is a budgeted experiment over that configuration space, not an unbounded search for the prettiest validation score.

Understand defaults and ranges first

Record the library default as a baseline and know which direction increases complexity. Regularization often needs a log-spaced range across orders of magnitude, while depth may need only a few integers. Dense grids of meaningless decimals waste compute and create opportunities to fit validation noise.

Grid search

Grid search evaluates every declared combination. It is easy to explain when two or three knobs each have a small candidate set, but combinations grow exponentially and spend equal effort on dimensions that may not matter.

Random search

Random search samples from declared distributions and can cover influential dimensions more broadly under a fixed trial budget. Use log-uniform distributions when ratios matter, bounded integers for depth, and conditional spaces when one choice activates another. Store the seed and every sampled configuration.

Bayesian optimization and Optuna

When each evaluation is expensive, optimization tools use previous trials to propose promising settings and can prune weak runs. They do not remove objective noise or cross-validation cost. Predeclare initial random trials, search ranges, and stopping rules so the optimizer does not become an efficient validation-noise hunter.

The number of trials is also a hyperparameter

After 100 trials on the same validation set, the selected maximum becomes optimistic even if no estimator fitted those rows directly. Use nested CV when an unbiased comparison is needed or preserve a final test set. Declare compute budget, maximum trials, and stopping conditions before searching.

Choose a stable plateau, not a fragile peak

Plot mean score and fold variability against important knobs. A broad region within noise of the best score is safer than one point on a cliff. Include training time, prediction time, and artifact size, freeze the selection, refit by the documented policy, and only then open test data.

Code

RandomizedSearchCV across a wide space·python

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

param_dist = {
    "clf__C": loguniform(1e-3, 1e3),
    "clf__l1_ratio": [0.0, 0.2, 0.5, 0.8, 1.0],
}
search = RandomizedSearchCV(
    pipe, param_distributions=param_dist, n_iter=40, cv=5,
    scoring="average_precision", random_state=7, n_jobs=-1
)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

Optuna for tree ensembles·python

import optuna
import lightgbm as lgb
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 800),
        "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3, log=True),
        "num_leaves": trial.suggest_int("num_leaves", 16, 256),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
    }
    model = lgb.LGBMClassifier(**params)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="average_precision")
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=40)