Transfer Learning vs Fine-Tuning

Three escalating commitments

Frozen feature extractor — backbone is frozen, only train a new head. Cheap, fast, often surprisingly competitive on small datasets. Always your first try.

Partial fine-tuning — unfreeze the last few layers + head, freeze the early layers. Compromise between capacity and overfitting risk on small data.

Full fine-tuning — train every parameter. Highest accuracy ceiling, highest compute cost, biggest overfitting risk. Use a much lower learning rate (1e-5 or 1e-6) than from-scratch training.

Tip: Always run the frozen baseline first. The number it gives you is the floor any more expensive approach has to beat. Many production projects ship with the frozen baseline because the gap to fully-fine-tuned wasn't worth the engineering cost.

The discriminative learning rate trick

Use a higher learning rate for the new head (it's random and needs more updates) and a lower learning rate for the pretrained backbone (it's already good and large updates would damage it). Common ratio: 10x to 100x.

When fine-tuning hurts

Small dataset + full fine-tune = overfitting paradise. The pretrained backbone has 100M+ parameters and your dataset has 1000 examples — the model will memorize them in ten epochs and degrade on the test set. Stay frozen, use stronger augmentation, or escalate gradually.

Principle: Fine-tuning is always the right escalation order: frozen → partial → full. Skipping straight to 'full fine-tune with default lr' is how teams discover their dataset is too small the painful way.

Code

Frozen, partial, and full fine-tune in one file·python

import torch.nn as nn
import torchvision.models as tvm
from torchvision.models import ResNet50_Weights

backbone = tvm.resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)

def freeze_all(model):
    for p in model.parameters(): p.requires_grad = False

def freeze_except(model, prefixes):
    for name, p in model.named_parameters():
        p.requires_grad = any(name.startswith(prefix) for prefix in prefixes)

# (1) Frozen feature extractor
freeze_all(backbone)
backbone.fc = nn.Linear(backbone.fc.in_features, 7)

# (2) Partial: only last block + head trainable
freeze_except(backbone, prefixes=["layer4", "fc"])

# (3) Full fine-tune with discriminative lr
for p in backbone.parameters(): p.requires_grad = True
opt = torch.optim.AdamW([
    {"params": [p for n, p in backbone.named_parameters() if not n.startswith("fc.")],
     "lr": 1e-5},
    {"params": [p for n, p in backbone.named_parameters() if n.startswith("fc.")],
     "lr": 1e-3},
])

Three escalating commitments

The discriminative learning rate trick

When fine-tuning hurts

Code

External links

Exercise

Progress

Comments 0