The Transformer's most surprising property is that it doesn't really care what its input represents. Once you turn a domain into a sequence of embeddings, the same attention machinery that learns English grammar will learn image structure, audio spectrograms, or protein folding rules.
Three concrete examples that have already shipped at scale:
- Vision Transformers (ViT, 2020). Split an image into 16×16 patches, linearly project each patch into an embedding, and feed the patch sequence through a Transformer encoder. ViT-H/14 reaches 88.5% top-1 on ImageNet. DINOv2 produces frozen visual features that LLaMA 3.2 vision and many multimodal LLMs reuse.
- Whisper (audio). An encoder-decoder Transformer where the encoder consumes log-mel spectrograms (audio frames) and the decoder generates text via cross-attention. Same unit cell, different tokenization at the input.
- AlphaFold 2 / ESMFold (biology). Variants of the Transformer applied to amino-acid sequences, predicting 3D protein structures. The attention matrix turns out to be a natural place to encode pairwise residue interactions.
The lesson: the Transformer is not a language architecture that happens to work on other things. It's a sequence-of-embeddings architecture. Anything you can tokenize fits.