What Is a Test Dataset?

The dataset is the ceiling

An eval suite is only as good as the dataset it runs on. The grader can be perfect; the framework can be best-in-class; the judge can be GPT-5. None of it matters if the inputs do not represent the real workload.

A test dataset is a deliberately curated collection of inputs, optionally paired with references (what good looks like) and metadata tags (so you can slice the suite later). The structure looks small. The discipline of building it is the actual job.

Three datasets, three roles

Smoke set — 5-20 cases that exercise the happy path. Runs in seconds. Catches the obvious "I broke the pipeline" bugs before you push.
Regression set — 100-500 cases that cover the behaviors you have committed to protecting. Runs on every PR. Catches silent quality regressions.
Adversarial / edge-case set — Hard, weird, malicious, or low-frequency cases. Runs less often. Catches the failures that wreck users when they hit them.

Principle: Coverage matters more than size. A 200-case dataset that covers 20 distinct behaviors beats a 5,000-case dataset that mostly tests one happy path.

Where the data comes from

Production logs — sample your real users (with privacy controls). The most representative source.
Domain experts — they know the failure modes you have not encountered yet.
Synthetic generation — LLMs can fabricate inputs at scale. Useful for adversarial sets and rare combinations.
Public benchmarks — useful for foundational capability checks, never sufficient for product-specific evals.

Code

JSONL dataset format — the canonical eval shape·json

{"id": "qa.001", "input": "What is the capital of France?", "reference": "Paris", "tags": ["qa", "easy", "english"]}
{"id": "qa.002", "input": "Who wrote Hamlet?", "reference": "William Shakespeare", "tags": ["qa", "easy", "english"]}
{"id": "qa.003", "input": "Capital of South Sudan?", "reference": "Juba", "tags": ["qa", "medium", "geography"]}
{"id": "qa.004", "input": "En quelle année est mort Napoléon?", "reference": "1821", "tags": ["qa", "french", "date"]}

Loading and stratifying for runs·python

import json
from collections import Counter

def load_jsonl(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

dataset = load_jsonl("datasets/regression.jsonl")
print(f"loaded {len(dataset)} cases")
print("tag distribution:", Counter(t for case in dataset for t in case["tags"]).most_common())

# Stratify — sample evenly across tags rather than picking the first 50
def stratified_sample(dataset, n_per_tag=10):
    by_tag = {}
    for case in dataset:
        for tag in case["tags"]:
            by_tag.setdefault(tag, []).append(case)
    out = []
    seen = set()
    for tag, cases in by_tag.items():
        for c in cases[:n_per_tag]:
            if c["id"] not in seen:
                out.append(c)
                seen.add(c["id"])
    return out

What Is a Test Dataset?

The dataset is the ceiling

Three datasets, three roles

Where the data comes from

Code

External links

Exercise

Progress

Comments 0