Encoder-Only: BERT, RoBERTa, and the Embedding Era

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al., 2018) showed that pre-training a Transformer encoder with masked language modeling produces representations powerful enough to push state of the art on every NLU benchmark with task-specific fine-tuning.

Architecture: 12 (base) or 24 (large) layers; d_model 768 or 1024; 12 or 16 heads; vocab 30,522 (WordPiece). Pre-training: masked LM (predict 15% randomly hidden tokens) plus Next Sentence Prediction (later shown to be unnecessary). Trained on BookCorpus + Wikipedia, ~3.3B tokens.

Why encoder-only fell out of favor for LLMs

Bidirectional attention is great for understanding but bad for generation — every token sees future tokens, so you can't sample autoregressively. By 2022 the field consolidated around decoder-only models that handle both understanding and generation through a single causal stack. BERT's children survive in two strongholds:

Embedding models for retrieval. BGE, E5, gte, mxbai-embed — all encoder-only Transformers fine-tuned to produce sentence embeddings for vector search. Critical infrastructure for RAG.
Classification heads. When you need to call a fine-tuned classifier 100M times a day on short inputs, a 110M-parameter BERT-base is dramatically cheaper than a 7B-parameter LLM and often more accurate.

Code

BERT for classification·python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

inputs = tok("The model is helpful.", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
print(torch.softmax(logits, dim=-1))
# After fine-tuning, this gives a 2-class probability.

Sentence embeddings with E5·python

from sentence_transformers import SentenceTransformer

# Modern encoder-only embedding model
m = SentenceTransformer("intfloat/multilingual-e5-large-instruct")
emb = m.encode([
    "Transformers changed AI.",
    "트랜스포머가 AI를 바꿨다.",
])
# emb is (2, 1024) — embeddings live in the same space across languages.

Encoder-Only: BERT, RoBERTa, and the Embedding Era

Why encoder-only fell out of favor for LLMs

Code

External links

Exercise

Progress

Comments 0