Judge Prompt Design

The judge prompt is the rubric in code

A judge is only as good as its prompt. A vague prompt produces a vague judge. The discipline of writing a good judge prompt is identical to the discipline of writing a good annotation rubric — same five elements, just rendered for an LLM instead of a human.

Five elements of a strong judge prompt

Role and stance — "You are an expert evaluator focusing on factual correctness." Not "be a judge."
Exact criteria — what passes, what fails, in plain language with no ambiguity.
Worked examples — at least one pass and one fail, with the verdict and the reasoning. Few-shot dramatically tightens calibration.
Edge case handling — what to do with refusals, partial answers, off-topic correct answers.
Strict output format — JSON with named fields, so parsing is robust.

The rationale-before-verdict trick

Ask the judge to think step-by-step before producing the verdict. Models reason better when they verbalize before deciding. Concretely: structure the JSON so "reasoning" appears before "verdict". This single change typically lifts judge accuracy 3-8 points.

Principle: A judge that thinks out loud — in the right order — judges better. Always put rationale before verdict in the output schema.

Few-shot examples are not optional for hard tasks

For nuanced criteria (faithfulness, helpfulness, tone), zero-shot judges drift. Three to five worked examples — including disagreement cases ("here is an output that LOOKS pass but is actually fail because...") — lock the judge to your rubric.

Temperature 0 for judges

Judges should be deterministic given the same input. Set temperature=0 and, if available, fix the seed. Otherwise the same eval run on Tuesday and Wednesday gives different numbers, and you cannot tell change from noise.

Code

Strong judge prompt — full skeleton·python

JUDGE_PROMPT = """
You are an expert evaluator measuring whether assistant responses are faithful to the provided context.

## Criteria
- PASS: every factual claim in the response is supported by the context.
- FAIL: the response contains at least one factual claim not supported by the context.
- Style, tone, length, and grammar do NOT influence the verdict.
- A refusal to answer is FAIL only if the context contains the answer.

## Examples

Example 1
Context: \"The Eiffel Tower is 330m tall and located in Paris.\"
Question: \"How tall is the Eiffel Tower?\"
Response: \"The Eiffel Tower is 330m tall.\"
Verdict: PASS
Reasoning: The 330m claim is directly supported by the context.

Example 2
Context: \"The Eiffel Tower is 330m tall and located in Paris.\"
Question: \"How tall is the Eiffel Tower?\"
Response: \"The Eiffel Tower is 324m tall, including its antenna.\"
Verdict: FAIL
Reasoning: 324m and the antenna detail are not in the context.

## Now evaluate this case
Context: {context}
Question: {question}
Response: {response}

Reply ONLY with JSON in exactly this shape (reasoning FIRST):
{{\"reasoning\": \"<one or two sentences>\", \"verdict\": \"PASS\" or \"FAIL\"}}
"""

Robust JSON parsing — judges sometimes drift·python

import json, re

def parse_judge_output(raw):
    raw = raw.strip()
    # Strip ```json fences if the judge added them
    fenced = re.search(r'```(?:json)?\s*(\{.*\})\s*```', raw, flags=re.S)
    if fenced:
        raw = fenced.group(1)
    try:
        d = json.loads(raw)
        if d.get("verdict") not in ("PASS", "FAIL"):
            raise ValueError("unexpected verdict")
        return d
    except Exception as e:
        return {"verdict": "FAIL", "reasoning": f"unparseable judge output: {e}"}

The judge prompt is the rubric in code

Five elements of a strong judge prompt

The rationale-before-verdict trick

Few-shot examples are not optional for hard tasks

Temperature 0 for judges

Code

External links

Exercise

Progress

Comments 0