Pointwise: present one output, ask the judge to score it 1-5 (or pass/fail).
Pairwise: present two outputs side-by-side, ask the judge to pick a winner (or call a tie).
Both are useful, in different situations. Many teams default to pointwise because it feels simpler; in practice pairwise is often more reliable.
Why pairwise is easier for LLMs
Absolute scoring requires calibration ("what does a 4 vs 5 mean?"). LLMs are notoriously bad at this — they cluster scores around the middle of the scale and rarely use the extremes. Pairwise sidesteps the problem: "is A better than B?" is an easier question than "rate A on a 1-5 scale."
Position bias and how to defeat it
Show two outputs to a judge and the judge will pick whichever appeared first more often than chance. Defense: every pair, run twice — once with A first, once with B first. Average the verdicts. Disagreement between orderings means the judge is too biased on this comparison; mark as TIE.
Principle: Always run pairwise comparisons in both orderings. Position bias is real and large enough to flip rankings if you ignore it.
When pairwise is the wrong tool
You need an absolute number, not a ranking (regulatory reports, SLAs).
You are evaluating one system over time, not comparing two systems.
You have many candidates (N=20). Pairwise needs N(N-1)/2 comparisons; pointwise just needs N.
Use pairwise to ground pointwise
A common pattern: run pairwise on a small set, derive an Elo-style ranking, and use the ranking to anchor pointwise calibration ("this is what a 5 looks like; this is what a 3 looks like"). It costs one evaluation cycle but produces a much more honest pointwise judge.
Code
Pairwise judge with position-bias defense·python
def pairwise_judge(question, output_a, output_b, judge_model):
def ask(first_label, first, second_label, second):
prompt = f"""
Question: {question}
Response {first_label}: {first}
Response {second_label}: {second}
Which response is better? Reply with JSON: {{\"reasoning\": \"...\", \"winner\": \"{first_label}\" or \"{second_label}\" or \"TIE\"}}"""
return parse_judge_output(judge_model.complete(prompt, temperature=0))
v1 = ask("A", output_a, "B", output_b)
v2 = ask("A", output_b, "B", output_a) # roles swapped
# Map v2 back: if v2 says A wins, that means output_b won on the second call.
win1 = v1["winner"]
win2 = {"A": "B", "B": "A", "TIE": "TIE"}[v2["winner"]]
if win1 == win2:
return win1, [v1["reasoning"], v2["reasoning"]]
return "TIE", [v1["reasoning"], v2["reasoning"]]
Pointwise scoring with explicit rubric anchors·python
POINTWISE_PROMPT = """
Score this response from 1 to 5.
5 = perfect: factually correct, complete, well-structured, no issues
4 = good: correct and complete, minor stylistic issues
3 = acceptable: mostly correct, but missing a notable detail or has a small error
2 = poor: major missing detail or factual error
1 = bad: largely incorrect, off-topic, or unhelpful
Question: {question}
Response: {response}
Reply with JSON: {{\"reasoning\": \"...\", \"score\": 1|2|3|4|5}}
"""
# Without anchors, judges cluster around 4. With anchors, distribution spreads.
Run a 20-case pairwise eval comparing two prompt versions in your product, using the position-bias defense. Compute the win rate. Now run a pointwise eval on the same set. Report which one gave you a clearer signal about which prompt to ship.
Progress
Progress is local-only — sign in to sync across devices.