What are you comparing the output to?
Reference-based evaluation provides the judge with a known-good answer (or a list of acceptable answers). The judge's job is to compare. Reference-free evaluation gives the judge only the question and the output. The judge must judge in absolute terms, often with a written rubric.
When to use each
| Reference-based | Reference-free |
|---|---|
| Translation, classification, RAG with ground truth | Open-ended generation, summarization, dialogue |
| You can build a golden dataset | Many valid outputs exist; building one golden answer is impossible |
| Tighter, more reproducible scoring | Looser, more rubric-dependent |
| Often combinable with deterministic graders (BLEU, BERTScore) | Almost always requires LLM judge |
Hybrid: reference-based with multiple references
For tasks where many phrasings are correct ("how to express this idea"), provide a list of acceptable references and let the judge accept if the output is functionally equivalent to any of them. This sits between strict reference-based and fully reference-free.
The rubric is your stand-in for the reference
In reference-free evaluation, the judge has nothing to anchor against except the rubric. The discipline of writing a tight rubric — exact criteria, edge cases, examples — matters even more than in reference-based mode. A vague rubric in reference-free mode produces a coin-flip judge.