Evaluation Methods

~20 min · evaluation, perplexity, llm-judge, metrics

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Four evaluation approaches

1. Perplexity

How surprised the model is by the test data. Lower = better. Good for sanity checks during training, but doesn't tell you about task quality.

2. Task-specific metrics

Accuracy, F1, BLEU, ROUGE — depends on your task. Classification → accuracy. Summarization → ROUGE. Generation quality → human preference rate.

3. LLM-as-judge

Use a strong model (GPT-4o, Claude) to evaluate outputs. Surprisingly reliable for many tasks and much cheaper than human evaluation.

4. Human evaluation

The gold standard. Have domain experts rate outputs on relevant dimensions (accuracy, helpfulness, safety, style). Expensive but irreplaceable for high-stakes use cases.

The combined evaluation strategy

Mix all four:

Perplexity for training-time monitoring.
Automated metrics for quick iteration.
LLM-as-judge for comparing model versions.
Human evaluation for final validation before deployment.

Code

LLM-as-judge for pairwise comparison·python

from openai import OpenAI

client = OpenAI()

def llm_judge(prompt: str, response_a: str, response_b: str) -> str:
    """Use GPT-4o to judge which response is better."""
    judgment = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": f"""Compare these two responses to the prompt below.

Prompt: {prompt}

Response A: {response_a}

Response B: {response_b}

Which response is better and why? Answer with "A" or "B" first, then a one-paragraph rationale focused on accuracy, helpfulness, and clarity.""",
        }],
        temperature=0,
    )
    return judgment.choices[0].message.content

# Tip: randomize A/B order between trials to remove position bias
import random
def fair_judge(prompt: str, base_resp: str, ft_resp: str) -> bool:
    if random.random() < 0.5:
        verdict = llm_judge(prompt, base_resp, ft_resp)
        return verdict.strip().upper().startswith("B")  # B = ft
    else:
        verdict = llm_judge(prompt, ft_resp, base_resp)
        return verdict.strip().upper().startswith("A")  # A = ft

External links

Exercise

Take a fine-tuned model and the base model. Build a 30-prompt eval set from your target use case. Run pairwise LLM-as-judge with randomized A/B positions. Report the win rate of fine-tuned vs base. Then sanity-check 5 random pairs by hand — does the judge agree with you?