Exact Match and Contains

The simplest graders, used way too rarely

Before you reach for an LLM judge, ask: "is there a string check that would settle this?" Most teams skip exact-match and contains because they feel beneath them. They are wrong. These graders are free, deterministic, instantaneous, and catch a surprising fraction of real regressions.

When exact-match works

Single-token answers (yes/no, true/false, A/B/C/D classification).
Numeric answers with a fixed format.
Code generation when the prompt asks for a specific function name.
Structured fields inside JSON output.

Why 'contains' is the workhorse

For most natural-language tasks, exact match is too strict ("Paris" vs "Paris is the capital of France"). The contains grader checks whether the reference appears anywhere in the output, optionally case-insensitively. It accepts wordy or terse phrasing while still catching outputs that are wrong.

Principle: Exhaust deterministic graders before paying for an LLM judge. They are 1000x cheaper and catch more regressions than people credit them for.

Multi-reference contains

Real questions often have multiple acceptable phrasings. "What is the capital of South Sudan?" can be answered "Juba", "Juba is the capital", or "South Sudan's capital is Juba." Storing a list of acceptable references and matching any one of them keeps the grader strict on substance and lenient on style.

Code

Exact-match and contains, three flavors·python

import re

def exact(output, reference):
    return output.strip() == reference.strip()

def contains(output, reference, case_insensitive=True):
    if case_insensitive:
        output, reference = output.lower(), reference.lower()
    return reference in output

def contains_any(output, references, case_insensitive=True):
    return any(contains(output, r, case_insensitive) for r in references)

def contains_all(output, references, case_insensitive=True):
    return all(contains(output, r, case_insensitive) for r in references)

# Usage
assert contains("The capital is Paris", "paris")            # True
assert contains_any("The capital is Juba", ["juba", "jouba"])
assert contains_all("Citizens of Tokyo and Osaka", ["Tokyo", "Osaka"])

Word-boundary contains — beware substrings·python

# 'cat' is in 'concatenate'. Naive contains says cats appear in your output.
# Use word boundaries when the reference is a short word.
import re

def contains_word(output, word, case_insensitive=True):
    flags = re.IGNORECASE if case_insensitive else 0
    pattern = rf"\b{re.escape(word)}\b"
    return re.search(pattern, output, flags) is not None

assert not contains_word("concatenate things", "cat")  # True — substring trap avoided
assert contains_word("the cat sat", "cat")              # True

Exact Match and Contains

The simplest graders, used way too rarely

When exact-match works

Why 'contains' is the workhorse

Multi-reference contains

Code

External links

Exercise

Progress

Comments 0