Hyperparameter Tuning

The patterns that separate a Keras tutorial alumnus from someone who ships. KerasTuner for hyperparameter sweeps, mixed-precision (FP16/BF16) for 2-3× speedup, multi-GPU distribution strategies, post-training quantization, knowledge distillation, and a debugging playbook for when training goes silently wrong.

Stop hand-tuning the dials

You can squeeze a few points of accuracy by guessing learning rates and layer widths over a long afternoon — but you're doing a blind, biased search with your gut as the optimizer. KerasTuner replaces that loop: you declare the search space once inside a build_model(hp) function, and the tuner runs the trials, tracks the scores, and hands you the best model back.

The shape of a search

The key move is that hyperparameters become arguments you sample, not constants you type. hp.Int("units", 32, 512, step=32) turns layer width into a tunable axis; hp.Float("lr", 1e-4, 1e-2, sampling="log") does the same for the learning rate — and sampling="log" matters, because learning rate lives on a logarithmic scale, so you want trials spread evenly across orders of magnitude, not bunched near the top of the range.

Pick a search strategy

RandomSearch — uniform sampling. The honest baseline; surprisingly hard to beat on small spaces.
BayesianOptimization — fits a Gaussian-process surrogate to past trials and samples where it expects improvement. Smart, but it assumes a fairly smooth landscape.
Hyperband — a bandit method that starts many trials cheaply and kills the weak ones early, pouring the saved budget into the survivors. Usually the right default for deep learning, where most configs are bad and you want to find that out fast.

Code

Define a search space and run BayesianOptimization·python

import keras_tuner

def build_model(hp):
    model = keras.Sequential([
        keras.Input(shape=(784,)),
        layers.Dense(
            units=hp.Int("units", min_value=32, max_value=512, step=32),
            activation="relu",
        ),
        layers.Dropout(hp.Float("dropout", 0.0, 0.5, step=0.1)),
        layers.Dense(10, activation="softmax"),
    ])
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Float("lr", 1e-4, 1e-2, sampling="log")
        ),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"],
    )
    return model

# Search strategies
tuner = keras_tuner.BayesianOptimization(
    build_model, objective="val_accuracy", max_trials=20,
)
tuner.search(x_train, y_train, epochs=10, validation_split=0.2)
best_model = tuner.get_best_models()[0]

Stop hand-tuning the dials

The shape of a search

Pick a search strategy

Code

External links

Exercise

Progress

Comments 0