Golden Datasets

The golden dataset is your ground truth

A "golden" dataset is one where each input is paired with a reference output that has been blessed by humans — domain experts, annotators, or product owners. It is the reference frame your graders compare against. Building one is slow, expensive, and irreplaceable.

What makes a dataset 'golden'

Verified references — at least two independent humans agreed on the correct output. Single-annotator references inherit one person's biases.
Edge case coverage — not just easy cases. Long inputs, adversarial inputs, cross-language inputs, format-shifting inputs.
Versioning — the dataset is checked into git (or a dataset platform) so you can tell when, why, and by whom each case changed.
Provenance — for each case, you know where it came from (production log, expert, synthetic) and can defend its inclusion.

Principle: A golden dataset is the most expensive thing you build and the one piece of eval infrastructure you cannot buy back later. Treat it like a database, not a scratch file.

Inter-annotator agreement matters

If two annotators disagree on the "correct" answer for 30% of cases, your golden dataset is not golden — it is a measurement of human disagreement. Compute Cohen's kappa or Krippendorff's alpha. If agreement is below ~0.7, your task definition is too vague; rewrite the rubric before continuing to label.

Beware leakage

If your golden set was used to fine-tune the model you are evaluating, your eval is meaningless — the model has memorized the answers. Always hold out a portion of the dataset that never touches training. This is harder than it sounds when teams casually pass datasets around.

Code

Golden dataset entry — beyond input + reference·json

{
  "id": "qa.012",
  "input": "What was the population of Tokyo in 2024?",
  "reference": "Approximately 14 million in the prefecture; 37 million in the wider metropolitan area.",
  "acceptable_alternatives": [
    "~14 million in the city, ~37 million metro",
    "14 million people lived in Tokyo in 2024"
  ],
  "tags": ["qa", "factual", "date-sensitive"],
  "annotators": ["alice", "bob"],
  "agreement": 1.0,
  "source": "production_log",
  "provenance": "sampled 2026-04-15, anonymized",
  "version": 3
}

Inter-annotator agreement check (Cohen's kappa)·python

from sklearn.metrics import cohen_kappa_score

# Two annotators rate 100 outputs on a binary 'correct or not'
alice = [1, 0, 1, 1, 0, 1, ...]  # length 100
bob   = [1, 0, 1, 0, 0, 1, ...]

kappa = cohen_kappa_score(alice, bob)
print(f"Cohen's kappa = {kappa:.3f}")
# < 0.4 — poor agreement, your task definition is fuzzy
# 0.4-0.6 — moderate, worth tightening rubric
# 0.6-0.8 — substantial, reasonable to label more
# > 0.8 — almost perfect, ready for production

The golden dataset is your ground truth

What makes a dataset 'golden'

Inter-annotator agreement matters

Beware leakage

Code

External links

Exercise

Progress

Comments 0