Pairwise Comparison — When Absolute Scoring Lies

~14 min · evaluation, pairwise

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Humans (and judges) compare better than they score

Asking "is this output good?" produces noisy 1-to-5 scores; asking "is A or B better?" produces tight, reliable comparisons. For subjective tasks (writing quality, helpfulness, tone), pairwise wins.

The pairwise eval shape

Run prompt v_old and v_new on the same N inputs.
For each input, present the two outputs side-by-side to a judge (human or LLM).
Judge picks A, B, or tie.
Aggregate: v_new wins X% of the time. Significant if X is meaningfully above 50%.

Pitfalls

Position bias — judges favor the first option. Randomize order.
Length bias — judges favor longer outputs. Add length-control instructions.
Verbosity bias — wordier outputs sound more thoughtful. Spell out criteria.
Self-preference — LLM judges tend to prefer outputs from the same model family.

Code

Pairwise judge prompt·markdown

## Task
You are comparing two answers (A and B) to the same support question. Pick the one that better satisfies the rubric.

## Rubric
- Accuracy: cites correct policy.
- Clarity: opens with the verdict.
- Tone: warm but terse, no apology preamble.
- Length: ≤ 100 words.

## Output
{"winner": "A" | "B" | "tie", "reason": "<one sentence>"}

## Question
{{q}}

## A
{{out_a}}

## B
{{out_b}}

External links

Exercise

Run pairwise comparison between your current prompt and a candidate revision on 50 inputs. Use an LLM judge with a clear rubric. Randomize order. Report the win rate.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousGolden Sets — Building and Maintaining Them Next →LLM-as-Judge — Use Cases and Pitfalls

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.