Dataset Size and Diversity

Diversity beats size, but you still need enough size

"How many cases do I need?" is the wrong question. The right question is: "do my cases represent every behavior I care about, with enough samples per behavior to detect the regression I care about?"

Statistical floor

For a binary pass/fail metric, detecting a 5% pass-rate change with 95% confidence requires roughly 400 cases. Detecting a 10% change requires ~100. If you only have 20 cases, your suite cannot tell a real regression from noise — even when the average shifts by 10 percentage points.

The diversity dimensions

Topic / domain — coding, medical, legal, casual chat all have different failure modes.
Difficulty — easy, medium, hard, edge. A suite of all easy cases hides the regressions that matter.
Length — short inputs and long inputs trigger different model behaviors. Always include both.
Language / locale — at least sample your top 3 user languages.
User intent — informational, navigational, transactional, exploratory.
Time — sample across multiple days; a Monday-only sample misses weekend traffic.

Principle: A 100-case dataset that covers 10 dimensions evenly tells you more than a 1,000-case dataset that all looks alike.

Imbalance lies to you

If 80% of your dataset is "easy English chitchat," the average pass rate will reflect that and hide regressions in the other 20%. Always report per-tag pass rates alongside the aggregate. The aggregate is for executives; the per-tag breakdown is for engineers.

Code

Sample size for binary pass-rate detection·python

from math import ceil

def sample_size_for_difference(p1, p2, alpha=0.05, power=0.8):
    """Approximate two-proportion z-test sample size per group."""
    from statistics import NormalDist
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_beta  = NormalDist().inv_cdf(power)
    p_bar = (p1 + p2) / 2
    numerator = (z_alpha * (2 * p_bar * (1 - p_bar)) ** 0.5
                 + z_beta * (p1 * (1 - p1) + p2 * (1 - p2)) ** 0.5) ** 2
    return ceil(numerator / (p1 - p2) ** 2)

print(sample_size_for_difference(0.90, 0.85))  # ~462 — to detect a 5pt drop
print(sample_size_for_difference(0.90, 0.80))  # ~120 — to detect a 10pt drop

Per-tag pass rates beat the aggregate·python

from collections import defaultdict

def pass_rate_by_tag(results):
    by_tag = defaultdict(lambda: [0, 0])  # [passed, total]
    for r in results:
        for tag in r["tags"]:
            by_tag[tag][1] += 1
            if r["score"] >= 1.0:
                by_tag[tag][0] += 1
    return {tag: passed / total for tag, (passed, total) in by_tag.items()}

# Output:
# {"easy": 0.95, "hard": 0.62, "long": 0.71, "polish": 0.40}
# Aggregate said 88%. Polish is on fire.  ← that's the real story

Dataset Size and Diversity

Diversity beats size, but you still need enough size

Statistical floor

The diversity dimensions

Imbalance lies to you

Code

External links

Exercise

Progress

Comments 0