Transfer Learning — Feature Extraction vs Fine-Tuning

The most cost-effective trick in computer vision

Training a strong vision model from scratch needs millions of labeled images. Almost no real project has that. Transfer learning solves this by starting from a model pretrained on ImageNet (1.28M images, 1000 classes) and adapting it to your task with a fraction of the data.

Two phases, in this order:

Feature extraction — freeze the backbone, train only a new classifier head on your data. The pretrained features are good enough to give 80–90% of the eventual accuracy with very little compute.
Fine-tuning — unfreeze the backbone (or the last few blocks), use a much smaller learning rate (e.g., 1/10), and continue training. This nudges the pretrained features toward your specific task.

Always freeze before unfreezing. If you start with all weights trainable, the random classifier head produces huge gradients that destroy the pretrained backbone weights in a few steps. Always do feature extraction first, then fine-tune.

Code

Feature extraction phase·python

import tensorflow as tf
from tensorflow import keras

base_model = keras.applications.EfficientNetV2S(
    input_shape=(224, 224, 3),
    include_top=False,
    weights='imagenet',
)
base_model.trainable = False    # FREEZE backbone

inputs = keras.Input(shape=(224, 224, 3))
x = base_model(inputs, training=False)
x = keras.layers.GlobalAveragePooling2D()(x)
x = keras.layers.Dropout(0.3)(x)
outputs = keras.layers.Dense(NUM_CLASSES, activation='softmax')(x)

model = keras.Model(inputs, outputs)
model.compile(
    optimizer=keras.optimizers.Adam(1e-3),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'],
)
model.fit(train_ds, validation_data=val_ds, epochs=10)

Fine-tuning phase — 더 작은 LR로·python

# After feature extraction is converged, unfreeze (some of) the backbone
base_model.trainable = True

# Optional: freeze BatchNormalization layers — they tend to harm fine-tuning
for layer in base_model.layers[:100]:    # freeze first 100 layers
    layer.trainable = False

# Recompile with a much smaller LR — typically 10x smaller
model.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'],
)
model.fit(train_ds, validation_data=val_ds, epochs=10)

Transfer Learning — Feature Extraction vs Fine-Tuning

The most cost-effective trick in computer vision

Code

External links

Exercise

Progress

Comments 0