BLEU and ROUGE

Two old metrics, still relevant when used carefully

BLEU and ROUGE were invented in the early 2000s for machine translation and summarization. They are imperfect, often criticized, and yet still useful for the narrow tasks they were built for: when you have reference outputs and you want a fast, free, language-agnostic similarity score.

BLEU — for translation

BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between candidate output and one or more references, with a brevity penalty to discourage suspiciously short outputs. Scores range 0-1 (or 0-100 in some libraries). 0.3 is decent, 0.5 is good, 0.7+ is rare on real translation tasks.

ROUGE — for summarization

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is the summarization sibling. ROUGE-N measures n-gram overlap, ROUGE-L measures longest common subsequence, ROUGE-W weights consecutive matches. Higher recall = the candidate covers more of the reference.

Principle: BLEU and ROUGE measure surface overlap, not meaning. They reward outputs that share words with the reference. They punish creative paraphrases that say the same thing differently.

When NOT to use them

Open-ended generation (creative writing, brainstorming) — many valid outputs, low overlap with any single reference.
Code generation — different code can be functionally identical and lexically distant.
Dialogue — a good response can share zero words with a reference.
Anything where meaning matters more than surface form. Use BERTScore or LLM judges instead.

Code

BLEU with sacrebleu — the standard reference impl·python

# pip install sacrebleu
from sacrebleu import corpus_bleu, sentence_bleu

refs   = [["The cat sat on the mat."]]
hyps   = ["The cat is on the mat."]
bleu = sentence_bleu(hyps[0], refs[0])
print(bleu.score)   # ~46 (out of 100)

# Corpus-level — better statistics, what papers report.
refs_corpus = [["The cat sat on the mat.", "It is raining."]]
hyps_corpus = ["The cat is on the mat.", "It rains today."]
corpus = corpus_bleu(hyps_corpus, refs_corpus)
print(corpus.score, corpus.brevity_penalty)

ROUGE — ROUGE-1, ROUGE-2, ROUGE-L·python

# pip install rouge-score
from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)

reference = "The cat sat on the mat and watched the rain."
candidate = "A cat watched the rain from the mat."

result = scorer.score(reference, candidate)
for metric, score in result.items():
    print(f"{metric}: P={score.precision:.2f} R={score.recall:.2f} F1={score.fmeasure:.2f}")

# Output:
# rouge1: P=0.62 R=0.50 F1=0.56
# rouge2: P=0.40 R=0.30 F1=0.34
# rougeL: P=0.62 R=0.50 F1=0.56

Why BLEU misleads on real systems·python

# Two outputs that mean the same thing — BLEU disagrees.
ref  = "The doctor prescribed antibiotics for the infection."
a    = "The doctor prescribed antibiotics for the infection."   # identical
b    = "For the infection, the physician prescribed antibiotics."  # paraphrase

from sacrebleu import sentence_bleu
print(sentence_bleu(a, [ref]).score)  # ~100
print(sentence_bleu(b, [ref]).score)  # ~30 — penalized for word reordering

Two old metrics, still relevant when used carefully

BLEU — for translation

ROUGE — for summarization

When NOT to use them

Code

External links

Exercise

Progress

Comments 0