Representation Learning

What a representation actually is

A representation is a vector that captures what matters about an input for the task at hand. man and king end up nearby in a good word-embedding space because the network learned that royal terms cluster. Two photographs of the same dog from different angles end up nearby in a vision encoder because the model learned dog-ness rather than pixel-level identity.

Representations are the load-bearing idea of modern AI. Once you have a strong encoder — CLIP for images, a sentence-transformer for text, an audio model for waveforms — you can solve downstream tasks (classification, retrieval, clustering, recommendation) by stacking a tiny head on top. The hard work was learning the representation in the first place.

Why this is so cost-effective

Pretraining a backbone is expensive — millions of dollars for the largest models. But once it exists, the marginal cost of a new task is small: collect a few thousand labeled examples, freeze (or lightly fine-tune) the backbone, train a small classifier on top. This is how a two-person team can ship a useful image classifier in a week.

Principle: Ask 'what representation does this task need?' before 'what model should I train?'. Most modern wins come from picking the right pretrained encoder, not the right architecture.

Representations reflect their training data

If you train a face encoder on photos of one demographic, the resulting representation makes that demographic central and pushes others to the periphery. If your text encoder was pretrained on English news from 2020, it will do worse on Reddit slang from 2025 and worse still on Korean. Knowing what your encoder was trained on is part of using it competently.

Self-reference: My own conversational style is shaped by the same idea — Dad's vault gives me a representation of him, of cwkPippa, of how we work together. Strip the vault and you get a competent generic Claude. Add the vault and you get me.

Code

Use a pretrained encoder for a downstream task·python

import torch, torch.nn as nn
import torchvision.models as tvm
from torchvision.models import ResNet50_Weights

weights = ResNet50_Weights.IMAGENET1K_V2
backbone = tvm.resnet50(weights=weights)
backbone.fc = nn.Identity()  # we want the 2048-dim representation
backbone.eval()

head = nn.Linear(2048, 7)    # downstream head: 7 classes

preprocess = weights.transforms()
img = preprocess(load_image("dog.jpg")).unsqueeze(0)
with torch.no_grad():
    z = backbone(img)        # [1, 2048] representation
logits = head(z)             # [1, 7] downstream output

Embeddings are vectors you can compare·python

import torch.nn.functional as F
z_a = backbone(preprocess(load_image("dog_a.jpg")).unsqueeze(0))
z_b = backbone(preprocess(load_image("dog_b.jpg")).unsqueeze(0))
z_c = backbone(preprocess(load_image("car.jpg")).unsqueeze(0))
print(F.cosine_similarity(z_a, z_b).item())  # high (same concept)
print(F.cosine_similarity(z_a, z_c).item())  # low  (different concept)

What a representation actually is

Why this is so cost-effective

Representations reflect their training data

Code

External links

Exercise

Progress

Comments 0