Types of Evaluation

Six axes that classify any eval

Before you choose a framework or a metric, locate the eval on six axes. The combination tells you what to build.

Online vs offline. Online runs on live traffic; offline runs on a frozen dataset.
Reference-based vs reference-free. Reference-based has a known correct answer (translation, classification, retrieval); reference-free does not (summarization, generation, helpfulness).
Pointwise vs pairwise. Pointwise scores one output in isolation; pairwise compares two outputs and picks a winner.
Deterministic vs model-graded. Deterministic uses regex / exact-match / numerical metrics; model-graded uses an LLM-as-judge.
Black-box vs white-box. Black-box looks at final output only; white-box inspects intermediate steps (retrieved docs, tool calls, chain-of-thought).
Functional vs safety. Functional checks task quality; safety checks for toxicity, bias, jailbreaks, PII leakage, prompt injection.

Pick the combination that matches the question

"Did my prompt change make summaries better?" → offline, reference-free, pairwise, model-graded, black-box, functional.

"Is the live system getting jailbroken?" → online, reference-free, pointwise, model-graded, black-box, safety.

"Is the retriever finding the right docs?" → offline, reference-based, pointwise, deterministic, white-box, functional.

Principle: Locate the eval on the six axes before picking a tool. Most "this framework can't do what we need" complaints are really "we asked it to do two different evals at once."

Whitebox lets you debug, blackbox protects you from coupling

White-box evals (looking at retrieved chunks, tool-call sequences, draft thoughts) are essential for diagnosis but couple the eval to your implementation. Re-architect the system and the eval breaks. Black-box evals on the final output are stable across implementations but do not tell you why something failed. Real teams keep both: a small set of white-box probes for diagnosis, a large set of black-box checks for regression.

Code

Eval-axis tagging makes mixed suites navigable·python

# Tag each eval case so you can slice the suite by axis.
EVAL_CASE = {
    "id": "summarizer.long_doc.001",
    "input": {...},
    "reference": None,                  # axis: reference-free
    "axes": {
        "phase": "offline",
        "reference": "free",
        "comparison": "pointwise",
        "grader": "llm-judge",
        "surface": "blackbox",
        "category": "functional",
    },
    "tags": ["summarization", "long-document", "english"],
}

Pairwise vs pointwise — the same task, two questions·python

# Pointwise: 'is this output good?' (absolute scale)
def pointwise(output):
    return judge.score(
        f"Rate this answer on a 1-5 scale for accuracy and completeness:\n{output}"
    )

# Pairwise: 'which of these two is better?' (relative, less calibration burden)
def pairwise(output_a, output_b):
    verdict = judge.complete(
        f"Output A: {output_a}\n\nOutput B: {output_b}\n\nWhich is better, A or B? Answer A, B, or TIE."
    )
    return verdict.strip().upper()

Six axes that classify any eval

Pick the combination that matches the question

Whitebox lets you debug, blackbox protects you from coupling

Code

External links

Exercise

Progress

Comments 0