A judge is only as good as its prompt. A vague prompt produces a vague judge. The discipline of writing a good judge prompt is identical to the discipline of writing a good annotation rubric — same five elements, just rendered for an LLM instead of a human.
Five elements of a strong judge prompt
Role and stance — "You are an expert evaluator focusing on factual correctness." Not "be a judge."
Exact criteria — what passes, what fails, in plain language with no ambiguity.
Worked examples — at least one pass and one fail, with the verdict and the reasoning. Few-shot dramatically tightens calibration.
Edge case handling — what to do with refusals, partial answers, off-topic correct answers.
Strict output format — JSON with named fields, so parsing is robust.
The rationale-before-verdict trick
Ask the judge to think step-by-step before producing the verdict. Models reason better when they verbalize before deciding. Concretely: structure the JSON so "reasoning" appears before "verdict". This single change typically lifts judge accuracy 3-8 points.
Principle: A judge that thinks out loud — in the right order — judges better. Always put rationale before verdict in the output schema.
Few-shot examples are not optional for hard tasks
For nuanced criteria (faithfulness, helpfulness, tone), zero-shot judges drift. Three to five worked examples — including disagreement cases ("here is an output that LOOKS pass but is actually fail because...") — lock the judge to your rubric.
Temperature 0 for judges
Judges should be deterministic given the same input. Set temperature=0 and, if available, fix the seed. Otherwise the same eval run on Tuesday and Wednesday gives different numbers, and you cannot tell change from noise.
Code
Strong judge prompt — full skeleton·python
JUDGE_PROMPT = """
You are an expert evaluator measuring whether assistant responses are faithful to the provided context.
## Criteria
- PASS: every factual claim in the response is supported by the context.
- FAIL: the response contains at least one factual claim not supported by the context.
- Style, tone, length, and grammar do NOT influence the verdict.
- A refusal to answer is FAIL only if the context contains the answer.
## Examples
Example 1
Context: \"The Eiffel Tower is 330m tall and located in Paris.\"
Question: \"How tall is the Eiffel Tower?\"
Response: \"The Eiffel Tower is 330m tall.\"
Verdict: PASS
Reasoning: The 330m claim is directly supported by the context.
Example 2
Context: \"The Eiffel Tower is 330m tall and located in Paris.\"
Question: \"How tall is the Eiffel Tower?\"
Response: \"The Eiffel Tower is 324m tall, including its antenna.\"
Verdict: FAIL
Reasoning: 324m and the antenna detail are not in the context.
## Now evaluate this case
Context: {context}
Question: {question}
Response: {response}
Reply ONLY with JSON in exactly this shape (reasoning FIRST):
{{\"reasoning\": \"<one or two sentences>\", \"verdict\": \"PASS\" or \"FAIL\"}}
"""
Robust JSON parsing — judges sometimes drift·python
import json, re
def parse_judge_output(raw):
raw = raw.strip()
# Strip ```json fences if the judge added them
fenced = re.search(r'```(?:json)?\s*(\{.*\})\s*```', raw, flags=re.S)
if fenced:
raw = fenced.group(1)
try:
d = json.loads(raw)
if d.get("verdict") not in ("PASS", "FAIL"):
raise ValueError("unexpected verdict")
return d
except Exception as e:
return {"verdict": "FAIL", "reasoning": f"unparseable judge output: {e}"}
Take a judge prompt you currently use (or your draft from the previous lesson). Add three worked examples — at least one tricky case — and switch the JSON to put reasoning before verdict. Re-run on 30 cases and compare against human ratings.
Progress
Progress is local-only — sign in to sync across devices.