What Transfer Learning Actually Is

You don't have to start from scratch — usually you shouldn't

A model pretrained on a huge dataset (ImageNet for vision, the Common Crawl for text) has learned representations that generalize. Adapting that to your specific task — almost always — beats training a fresh model on your (much smaller) dataset, even when your task seems unrelated.

Two extremes, plus the middle

Feature extraction — freeze the entire pretrained model, train only a new head. Cheap, fast, needs little data. Works well when your task is "close enough" to the pretraining task.
Full fine-tuning — unfreeze everything, train the whole model on your data with a small learning rate. Best results, needs more data and care. Standard for serious work.
Partial fine-tuning — freeze early layers (which learn generic features) and unfreeze later layers (which learn task-specific features). The pragmatic middle.

The big modern addition: parameter-efficient fine-tuning

For huge models (BERT-large, Llama, ViT-Huge), even full fine-tuning is expensive — you need to update billions of parameters. LoRA (Low-Rank Adaptation) and friends inject tiny trainable matrices alongside the frozen weights, training as little as 0.1–1% of the parameters while matching full-fine-tune quality. We'll get to LoRA in a later lesson; for now, just know it exists and changes the economics.

The torchvision weights API — modern

The old models.resnet50(pretrained=True) is deprecated. The modern call is models.resnet50(weights=ResNet50_Weights.IMAGENET1K_V2). The new API gives you versioned weights AND weights.transforms() — the exact preprocessing the model was trained with. Always use it.

Code

Feature extraction — freeze backbone, train head·python

import torch
import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)

# Freeze the entire pretrained model
for p in model.parameters():
    p.requires_grad = False

# Replace the final classifier — new layer is trainable by default
num_classes = 5
model.fc = nn.Linear(model.fc.in_features, num_classes)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"trainable: {trainable:,} / {total:,} ({trainable/total*100:.2f}%)")
# trainable: 10,245 / 25,567,037 (0.04%)

Partial fine-tune — unfreeze later layers·python

import torch.nn as nn
from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)

# Freeze everything
for p in model.parameters():
    p.requires_grad = False

# Replace head + unfreeze layer4 (the deepest backbone block)
model.fc = nn.Linear(model.fc.in_features, 5)
for p in model.layer4.parameters():
    p.requires_grad = True

# Per-group learning rate — backbone gets smaller LR
import torch.optim as optim
optimizer = optim.AdamW([
    {'params': model.layer4.parameters(), 'lr': 1e-5},
    {'params': model.fc.parameters(),     'lr': 1e-3},
], weight_decay=0.01)

The model-specific transforms come from weights·python

from torchvision.models import resnet50, ResNet50_Weights

weights = ResNet50_Weights.IMAGENET1K_V2

# This is the EXACT preprocessing the model was pretrained with
preprocess = weights.transforms()
print(preprocess)
# ImageClassification(
#     crop_size=[224]
#     resize_size=[232]
#     mean=[0.485, 0.456, 0.406]
#     std=[0.229, 0.224, 0.225]
#     interpolation=InterpolationMode.BILINEAR
# )

# Pass it to your DataLoader transform pipeline. No more hard-coded means.

What Transfer Learning Actually Is

You don't have to start from scratch — usually you shouldn't

Two extremes, plus the middle

The big modern addition: parameter-efficient fine-tuning

The torchvision weights API — modern

Code

External links

Exercise

Progress

Comments 0