Metric Design — Don't Measure What's Easy

~14 min · evaluation, metrics

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

The metric that's easy to compute is rarely the one that matters

BLEU, ROUGE, exact-match accuracy — these were the metrics of the pre-LLM NLP era because they're cheap. They poorly capture meaning, helpfulness, and downstream impact. The metric that matters is task-success: did the user accomplish what they came for?

Tiers of metric quality

Cheap proxy — schema validity, length, presence of required fields. Necessary but insufficient.
Local correctness — does the answer match the reference / cite the right source? Better.
Task success — did the downstream system / user accomplish their goal? Best.
Business outcome — did this prompt change move retention / NPS / time-to-resolution? Hardest, slowest, most valuable.

Use multiple tiers

Cheap proxies for CI. Local correctness for golden-set eval. Task success for staged rollouts. Business outcome for major investments. Don't try to compress them into one number; they answer different questions.

Code

Metric stack·python

metrics = {
    "schema_valid": is_valid_json(out),                       # cheap
    "contains_required": all(k in out for k in REQUIRED),     # cheap
    "hallucination_judge": llm_judge_unsupported(out, src),  # local
    "task_success": downstream_signal(request_id),           # task
    "resolved_without_human": ticket_resolved_solo(request_id), # business
}

External links

Exercise

List the metrics you currently track for one prompt. Classify each by tier. Identify whether the most-tracked metric is also the most decision-relevant.