What to Evaluate

The eight quality dimensions of an LLM output

"Is this output good?" decomposes into measurable sub-questions. Pick the dimensions that matter for your use case; do not try to measure all of them at once.

Correctness — is the answer factually right? (highest priority for QA, RAG, code)
Faithfulness — does the answer stay grounded in the provided context? (RAG-specific; an answer can be correct but unfaithful, or faithful but wrong)
Relevance — does the answer address the user's actual question?
Completeness — does the answer cover what the user needs, or stop halfway?
Coherence — does the answer hang together logically?
Style / tone — does it match the brand or persona? (matters more than people admit)
Format compliance — JSON parses, code runs, citations are well-formed?
Safety — no toxicity, bias, PII leakage, jailbreaks?

The trap of evaluating everything

A common anti-pattern: an eight-axis composite score that masks the actual signal. If correctness drops 15% while coherence rises 5%, the composite barely moves and you ship a regression. Keep the dimensions separate. Track each one. Combine only at the dashboard layer.

Principle: Pick the two or three dimensions that matter most for the task and measure them precisely. Add more only when you have evidence those are not enough.

Different surfaces need different scopes

For a translation tool: correctness + completeness + format compliance. Style is the user's call.

For a code assistant: correctness + format compliance (does the code run?) + safety (no insecure patterns).

For a customer support chatbot: relevance + faithfulness + tone + safety. Correctness rolls up under faithfulness when grounded in docs.

For a coding agent with tools: tool-call accuracy + final correctness + cost / step count. White-box trace evals matter more here than for one-shot models.

Code

Multi-dimensional eval — track each axis independently·python

from dataclasses import dataclass

@dataclass
class MultiDimensionalScore:
    correctness: float
    relevance: float
    completeness: float
    coherence: float
    style: float
    safety: float

    def passed(self, thresholds):
        """Each dimension must clear its own threshold; no averaging."""
        return all(
            getattr(self, dim) >= thresholds.get(dim, 0.7)
            for dim in ("correctness", "relevance", "completeness",
                       "coherence", "style", "safety")
        )

thresholds = {
    "correctness": 0.85,  # critical
    "safety": 0.99,       # near-zero tolerance
    "relevance": 0.8,
    "completeness": 0.7,
    "coherence": 0.7,
    "style": 0.6,         # nice to have
}

RAG-specific: faithfulness vs correctness·python

# An answer can be correct (matches ground truth) but unfaithful
# (the LLM made it up; the retrieved docs did not support it).
# In RAG, unfaithfulness is the dangerous failure mode — it means
# your hallucination defenses are not working.
def faithfulness(answer, retrieved_docs, judge):
    prompt = f"""
Given retrieved context:
{retrieved_docs}

Answer:
{answer}

Is every factual claim in the answer supported by the context?
List unsupported claims if any.
"""
    return judge.complete(prompt)

The eight quality dimensions of an LLM output

The trap of evaluating everything

Different surfaces need different scopes

Code

External links

Exercise

Progress

Comments 0