Transfer Learning in Vision

Why transfer learning is the default

A pretrained backbone has already learned 'how images work' from millions of examples. Your downstream task — say, classify 7 species of bird — has maybe 5000 labeled examples. Training a CNN from scratch on 5000 images would overfit catastrophically. Transfer learning lets you reuse the backbone's understanding and just train a small head on your task.

Three flavors: frozen backbone (only train the head; cheapest, often surprisingly competitive), full fine-tune (train everything; best accuracy, more compute), partial unfreeze (train head + last few layers; middle ground).

Tip: Default to frozen backbone for your first run. It's fast, it gives you a baseline, and you might find it's enough. If accuracy is short, unfreeze the last block; if still short, full fine-tune with a tiny learning rate (1e-4 or 1e-5).

The lr trick for fine-tuning

For full fine-tuning, use a much smaller learning rate than you would for from-scratch training (1e-4 or 1e-5 instead of 1e-3). Optionally use a higher learning rate for the new head than the pretrained backbone — the head starts random and needs more updates than the backbone needs to adjust.

The data preprocessing match

Match the preprocessing the backbone was trained with. ImageNet pretrained models expect specific normalization (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) and a specific input size (usually 224×224). torchvision exposes the right transforms via weights.transforms() — use that, don't reinvent.

Principle: Transfer learning is the highest-leverage 'modeling' decision you can make for vision. A frozen ResNet50 + linear head usually beats anything you train from scratch on 10K images.

Code

Frozen-backbone transfer learning·python

import torch
import torch.nn as nn
import torchvision.models as tvm
from torchvision.models import ResNet50_Weights

weights = ResNet50_Weights.IMAGENET1K_V2
backbone = tvm.resnet50(weights=weights)

# Freeze all backbone parameters
for p in backbone.parameters():
    p.requires_grad = False

# Replace head with task-specific layer (only trainable)
backbone.fc = nn.Linear(backbone.fc.in_features, n_classes)

# Optimizer over only the trainable params
opt = torch.optim.AdamW([p for p in backbone.parameters() if p.requires_grad], lr=1e-3)

# Use the backbone's preprocessing
preprocess = weights.transforms()

Full fine-tune with discriminative learning rates·python

# Unfreeze everything but train head faster than backbone
for p in backbone.parameters():
    p.requires_grad = True

backbone_params = [p for n, p in backbone.named_parameters() if not n.startswith("fc.")]
head_params     = [p for n, p in backbone.named_parameters() if n.startswith("fc.")]

opt = torch.optim.AdamW([
    {"params": backbone_params, "lr": 1e-5},
    {"params": head_params,     "lr": 1e-3},
])

Why transfer learning is the default

The lr trick for fine-tuning

The data preprocessing match

Code

External links

Exercise

Progress

Comments 0