Chatbot Arena and MT-Bench

The benchmarks that actually correlate with user perception

Multiple-choice benchmarks measure narrow capabilities. Real users care about open-ended dialogue. Two benchmarks fill that gap.

Chatbot Arena (LMSYS 2023→ongoing)

Crowdsourced pairwise comparison: users see two anonymous model outputs, vote for the better one, and an Elo ranking emerges over millions of votes. The closest thing the field has to a "real user" benchmark. Updated continuously at lmarena.ai.

Why it works:

Real prompts from real users in the wild — distribution closer to product reality than fixed benchmarks.
Pairwise reduces calibration burden on raters.
Elo aggregation is robust to the diversity of judges.
Continuously updated, so contamination is much harder.

Limitations:

Skewed toward English and casual chat.
Voters are not domain experts — won't catch subtle factual errors.
Some categories (coding, long-context, reasoning) have separate sub-leaderboards because aggregate Elo dilutes them.

MT-Bench (Zheng et al. 2023)

80 multi-turn open-ended questions across 8 categories (writing, reasoning, math, coding, extraction, STEM, humanities, role-play). LLM-as-Judge (typically GPT-4) scores responses 1-10. Replaced largely by Arena, but still used in papers.

Principle: Chatbot Arena is the closest public benchmark to "what users actually feel." Use it as your default model-selection signal, then validate with product-specific evals.

Code

Reading Arena rankings·text

# Visit https://lmarena.ai/?leaderboard
#
# Filter by category for a more useful read:
#   Overall — general chat (English-skewed)
#   Hard Prompts — adversarial / niche prompts (better discriminator)
#   Coding — code-specific Elo
#   Math — multi-step reasoning Elo
#   Vision — multimodal Elo
#
# A 50-Elo gap is roughly the threshold where users notice differences.
# A 100-Elo gap is conspicuous quality difference.

Building your own pairwise + Elo evaluation·python

import math

def update_elo(winner_rating, loser_rating, k=32):
    expected = 1 / (1 + 10 ** ((loser_rating - winner_rating) / 400))
    return (
        winner_rating + k * (1 - expected),
        loser_rating - k * (1 - expected),
    )

ratings = {"model_a": 1500, "model_b": 1500, "model_c": 1500}

for judgment in pairwise_judgments:  # list of (winner_id, loser_id)
    w, l = judgment
    new_w, new_l = update_elo(ratings[w], ratings[l])
    ratings[w], ratings[l] = new_w, new_l

# After enough comparisons, ratings converge to a stable ranking.
# Same math the Arena uses, just on your own dataset and judges.

Chatbot Arena and MT-Bench

The benchmarks that actually correlate with user perception

Chatbot Arena (LMSYS 2023→ongoing)

MT-Bench (Zheng et al. 2023)

Code

External links

Exercise

Progress

Comments 0