BERTScore: Semantic Similarity

When meaning matters and exact words don't

BERTScore replaces n-gram overlap with embedding similarity. Each token in the candidate is matched to its most similar token in the reference using BERT (or any contextual embedding model), and the per-token similarities are averaged. Result: paraphrases that say the same thing in different words score high; lexical overlap with different meaning scores low.

How it works

Encode reference and candidate with a contextual model (BERT, RoBERTa, or modern equivalents).
For each token in the candidate, find its highest-similarity token in the reference.
Aggregate into precision (candidate-side), recall (reference-side), and F1.

Why it's better than BLEU/ROUGE for most tasks

BLEU treats "physician" and "doctor" as zero overlap. BERTScore treats them as nearly identical. The result correlates much more strongly with human judgment for paraphrase, summarization, and translation tasks.

Principle: When you need automated semantic comparison without paying an LLM, BERTScore is the workhorse. It captures meaning that surface metrics miss, at near-deterministic cost.

What it still won't catch

Logical errors — "X causes Y" and "Y causes X" can score high together.
Negations — "is safe" and "is not safe" share most embedding mass.
Numerical mistakes — "$1M" vs "$10M" looks semantically close to embeddings.

Practical considerations

Pick the model carefully. microsoft/deberta-large-mnli tends to outperform vanilla BERT for evaluation. Cache embeddings if you re-run on the same dataset. The first run is expensive; subsequent runs are basically free.

Code

BERTScore with the official library·python

# pip install bert-score
from bert_score import score

refs = [
    "The doctor prescribed antibiotics for the infection.",
    "The car would not start because the battery was dead.",
]
hyps = [
    "For the infection, the physician prescribed antibiotics.",
    "The vehicle failed to start due to a dead battery.",
]

# F1 is what most papers report
P, R, F1 = score(hyps, refs, lang="en", model_type="microsoft/deberta-large-mnli")
for h, f in zip(hyps, F1.tolist()):
    print(f"{f:.3f}  {h}")
# 0.94  For the infection, the physician prescribed antibiotics.
# 0.91  The vehicle failed to start due to a dead battery.
# Both >>>> what BLEU would have given (~0.30).

Cosine similarity from embeddings — the DIY version·python

# When you want fast embeddings without the BERTScore matching machinery.
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-mpnet-base-v2")

def cosine_sim(a, b):
    va, vb = model.encode([a, b])
    return float(np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb)))

print(cosine_sim("safe to drink", "potable"))     # ~0.78
print(cosine_sim("safe to drink", "not safe"))    # ~0.55  ← negation trap

BERTScore: Semantic Similarity

When meaning matters and exact words don't

How it works

Why it's better than BLEU/ROUGE for most tasks

What it still won't catch

Practical considerations

Code

External links

Exercise

Progress

Comments 0