Encoder-Decoder: T5, BART, Whisper

Encoder-decoder Transformers split the architecture into two halves: an encoder that consumes the input bidirectionally and a decoder that generates output autoregressively while attending to the encoder's representation via cross-attention. This shape was original to the 2017 paper and remains the right tool for tasks with a clean input-to-output mapping.

T5 (Raffel et al., 2019). "Text-to-Text Transfer Transformer." Frames every NLP task as text-to-text — translate, summarize, classify, answer all become "input → output" generation tasks. Pretrained with span corruption (mask contiguous spans, predict them).
BART (Lewis et al., 2019). Denoising autoencoder pretrained by corrupting text (masking, deletion, shuffling, rotation) and reconstructing. Strong on summarization and structured rewriting tasks.
Whisper (OpenAI, 2022). Encoder-decoder for speech recognition. Encoder consumes log-mel spectrograms (audio frames as patch tokens); decoder generates text via cross-attention. Multilingual, robust, open weights.
NLLB (Meta, 2022). No Language Left Behind. Encoder-decoder optimized for low-resource translation across 200+ languages.

For "talk to me" use cases, decoder-only has eaten encoder-decoder territory. But for translation, ASR, and well-defined structured generation tasks, encoder-decoder remains the right shape — the explicit two-stage structure aligns with the task.

Code

T5 inference·python

from transformers import T5Tokenizer, T5ForConditionalGeneration

tok = T5Tokenizer.from_pretrained("google/flan-t5-base")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")

# Prefix the task — T5's text-to-text framing
text = "translate English to Korean: The Transformer changed AI."
ids = tok(text, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

Encoder-Decoder: T5, BART, Whisper

Code

External links

Exercise

Progress

Comments 0