Judge Bias and Calibration

Judges have biases. Measure them.

An LLM judge is an instrument with known systematic errors. Treat it like a thermometer that reads two degrees high — usable, but only if you correct for the offset.

The four biases that matter

Position bias — in pairwise comparison, judges prefer first-shown (or sometimes last-shown) outputs. Defense: run both orderings, average.
Length bias — judges reward longer outputs even when shorter is better. Defense: include "Length should not influence the verdict" in the rubric; spot-check on truncated/expanded variants of the same answer.
Self-preference — judges prefer outputs that look like the model family they came from. Defense: use a different model family as the judge, or use multiple judges and look for agreement.
Verbosity / confidence bias — judges prefer assertive, polished outputs over hedged or uncertain ones, even when uncertainty is appropriate. Defense: explicit rubric clauses about appropriate hedging.

How to calibrate a judge

Sample 100 cases.
Have humans rate them.
Run the judge on the same 100 cases.
Compute Cohen's kappa or simple agreement rate.
Look at the disagreements. Update the prompt. Re-run.
Iterate until agreement reaches the same level as inter-human agreement.

Principle: A judge that has not been calibrated against humans on your specific task is a guess. Calibration is the difference between an instrument and a story.

Multiple judges → ensemble

For high-stakes evals, run 2-3 different judges (different model families) and aggregate. Disagreement among judges is a useful signal: those cases need human review. This is more expensive but dramatically more reliable than a single-judge pipeline.

Code

Calibration: measure agreement with humans·python

from sklearn.metrics import cohen_kappa_score

def calibrate(human_labels, judge_labels):
    """Both are lists of 0/1 (or PASS/FAIL)."""
    agree = sum(h == j for h, j in zip(human_labels, judge_labels)) / len(human_labels)
    kappa = cohen_kappa_score(human_labels, judge_labels)
    return {"agreement": agree, "kappa": kappa}

# Target: kappa > 0.7. Anything below 0.4 means your judge is barely
# better than random for this task.
# Inter-human agreement is the ceiling — judges rarely beat humans
# at agreeing with humans.

Length-bias diagnostic·python

# Quick experiment: take 30 outputs, ask the judge to rate them.
# Then ask it to rate truncated versions (50% length).
# If average score drops significantly, your judge has length bias.
import random

def length_bias_test(cases, judge):
    full_scores, short_scores = [], []
    for c in cases:
        out = c["output"]
        full_scores.append(judge(c, out))
        short = out[:len(out)//2] + "..."
        short_scores.append(judge(c, short))
    print(f"avg full: {sum(full_scores)/len(full_scores):.2f}")
    print(f"avg short: {sum(short_scores)/len(short_scores):.2f}")
    # Big gap → judge rewards length more than content.

Two-judge ensemble — agreement gate·python

def ensemble_judge(case, judge_gpt, judge_claude):
    a = judge_gpt(case)
    b = judge_claude(case)
    if a["verdict"] == b["verdict"]:
        return {"verdict": a["verdict"], "agreement": True}
    return {"verdict": "REVIEW", "agreement": False, "a": a, "b": b}

# Cases marked REVIEW go to a human queue. Cheap insurance for high-stakes evals.

Judge Bias and Calibration

Judges have biases. Measure them.

The four biases that matter

How to calibrate a judge

Multiple judges → ensemble

Code

External links

Exercise

Progress

Comments 0