Vision Transformers

Transformers eat vision (2020)

The Vision Transformer (ViT) breaks an image into a grid of small patches (e.g. 16×16), embeds each patch as a vector, and feeds the sequence of patch embeddings into a standard transformer encoder. No convolutions. Pretrained on enough data, ViT matches or beats CNNs across vision benchmarks.

This was a shock. The CNN inductive biases (locality, translation invariance) had been treated as essential for vision. ViT showed that with enough data and compute, attention can learn those biases instead of having them baked in.

Tip: ViT works best with large pretraining datasets (ImageNet-21k, JFT-300M). On small datasets (ImageNet-1k from scratch), CNNs still often win because their inductive biases are useful regularizers when data is scarce.

The patches → tokens trick

A 224×224 image with 16×16 patches gives 196 patches. Each patch is flattened to a 768-d vector via a linear projection (or, equivalently, a Conv2d with kernel=16, stride=16). Add a learned position embedding and a [CLS] token. Now you have a sequence of 197 tokens — feed it into a standard transformer encoder.

Modern vision foundation models

ViT is the architecture; the weights people actually use are pretrained foundation models: CLIP (OpenAI; image-text contrastive), DINOv2 (Meta; self-supervised), SAM (Meta; segmentation), SigLIP (Google; image-text). These are downloadable backbones that turn vision tasks into "embed the image, train a head" — far easier than training a network from scratch.

Principle: In 2026, your first move for any vision task is: pick a pretrained foundation model (DINOv2 for general vision, CLIP for vision-language, SAM for segmentation), embed your data, train a small head. Architecture choices matter much less than backbone choices.

Code

ViT in five lines via torchvision·python

import torchvision.models as tvm
from torchvision.models import ViT_B_16_Weights

weights = ViT_B_16_Weights.IMAGENET1K_V1
vit = tvm.vit_b_16(weights=weights)
preprocess = weights.transforms()
# vit takes [B, 3, 224, 224], outputs [B, 1000] logits

Use DINOv2 as a frozen vision backbone·python

import torch

# DINOv2 — strong self-supervised vision backbone from Meta
backbone = torch.hub.load("facebookresearch/dinov2", "dinov2_vitb14")
backbone.eval()
for p in backbone.parameters():
    p.requires_grad = False

# Forward — outputs 768-d global embedding per image
img = preprocess(pil_image).unsqueeze(0)        # [1, 3, 224, 224]
with torch.inference_mode():
    z = backbone(img)                           # [1, 768]

Transformers eat vision (2020)

The patches → tokens trick

Modern vision foundation models

Code

External links

Exercise

Progress

Comments 0