Vision Transformers and Multimodal Models

Vision Transformers (ViT, Dosovitskiy et al., 2020) treat an image as a sequence of patches and process it with a standard Transformer encoder. The breakthrough wasn't a new architecture — it was the demonstration that the same machinery works on images.

The patching pipeline

Reshape a 224×224 image into 14×14 patches of 16×16 pixels each.
Flatten each patch and project linearly into d_model.
Prepend a learnable [CLS] token; add positional embeddings.
Run through the Transformer encoder.

That's the entire architecture from the original paper. ViT-H/14 (632M params) reaches 88.5% top-1 on ImageNet. DINOv2 trained ViTs with self-supervised learning and produced visual features now reused across multiple multimodal LLMs (Llama 3.2 vision, many open chat-vision models).

Multimodal LLMs

Modern multimodal models pipe image patches through a vision encoder, project the result into the LLM's embedding space, and feed those "visual tokens" into the same decoder-only stack that processes text. The architecture is unchanged; what changes is the token alphabet at the input.

Llama 3.2-vision (11B / 90B): adds a vision encoder that produces visual tokens consumed by the same Llama 3 backbone.
GPT-4o, Gemini 2.5: native multimodal — text/image/audio share the residual stream from training onward.
Llama 4 Scout/Maverick: early-fusion multimodal with the MetaCLIP vision encoder, supporting 200 languages.

Code

ViT in 30 lines·python

import torch.nn as nn
from einops import rearrange

class ViT(nn.Module):
    def __init__(self, img_size=224, patch=16, d_model=768,
                 n_layers=12, n_heads=12, n_classes=1000):
        super().__init__()
        self.patch_proj = nn.Conv2d(3, d_model, patch, patch)  # patchify
        self.cls = nn.Parameter(torch.zeros(1, 1, d_model))
        self.pos = nn.Parameter(torch.zeros(1, (img_size // patch)**2 + 1, d_model))
        self.blocks = nn.ModuleList([
            TransformerBlock(d_model, MultiHeadAttention(d_model, n_heads),
                             FeedForward(d_model, 4 * d_model))
            for _ in range(n_layers)
        ])
        self.head = nn.Linear(d_model, n_classes)
    def forward(self, x):
        x = self.patch_proj(x)              # (B, d, H/p, W/p)
        x = rearrange(x, 'b d h w -> b (h w) d')
        cls = self.cls.expand(x.size(0), -1, -1)
        x = torch.cat([cls, x], dim=1) + self.pos
        for blk in self.blocks:
            x = blk(x)
        return self.head(x[:, 0])           # use [CLS] token's output

Vision Transformers and Multimodal Models

The patching pipeline

Multimodal LLMs

Code

External links

Exercise

Progress

Comments 0