Fine-Tuning

Step 2: unfreeze the top, retrain with a tiny learning rate

Once the head is trained and your accuracy has settled, fine-tuning squeezes out the rest. You flip base_model.trainable = True and then re-freeze everything except the last handful of layers. Why only the top? The early layers hold the universal edge/texture detectors you want to keep; the later layers hold the ImageNet-specific concepts that are worth gently bending toward your task.

The learning rate is the whole game

The single most important number here is the learning rate, dropped to roughly 1e-5 — about 100× lower than normal training. Your pretrained weights are already near a good solution; a normal-sized LR would take huge steps and wreck them (catastrophic forgetting). The tiny LR nudges them just enough to specialize without unlearning. Fine-tune for ~10–20 more epochs and watch validation accuracy — if it stops improving or starts overfitting, stop. The Code section walks the exact freeze/recompile/fit sequence.

Code

Unfreeze the top layers and fine-tune at 1e-5·python

# Unfreeze the top layers of the base model
base_model.trainable = True

# Freeze everything except the top 20 layers
for layer in base_model.layers[:-20]:
    layer.trainable = False

# CRITICAL: Re-compile with a very low learning rate
model.compile(
    optimizer=keras.optimizers.Adam(1e-5),  # 100x lower!
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)

# Continue training (~10-20 more epochs)
model.fit(train_ds, epochs=20, validation_data=val_ds)

Step 2: unfreeze the top, retrain with a tiny learning rate

The learning rate is the whole game

Code

External links

Exercise

Progress

Comments 0