Annotation Guidelines and Versioning

The annotation rubric is the contract

"Is this output good?" is not a question annotators can answer. "Does this output state the correct answer in fewer than 50 words, in formal English, citing at least one source?" is. The difference between those two prompts is the difference between a useful golden dataset and a folder of disagreements.

What a good rubric contains

The exact question the annotator answers — phrased as binary or short scale.
Worked examples of pass and fail outputs, with explanations.
Edge case decisions — what to do with empty outputs, refusals, partial answers, off-topic but correct answers.
Out-of-scope clauses — what should NOT influence the rating.
Calibration set — 10-20 cases every annotator labels first; their agreement on these proves they understand the rubric.

Principle: If two annotators disagree, the rubric is wrong, not the annotators. Rewrite until agreement.

Versioning the dataset like code

Datasets evolve. Every change — new case added, label corrected, edge case introduced — should be a commit with a message. Three structural choices, all valid:

Plain git — JSONL files in the repo, reviewed via PRs. Simplest, scales to ~10K cases.
DVC / lakeFS — git for large datasets, with diffing tooling.
Platform-managed — Braintrust, Hugging Face Datasets, Argilla. Versioning + UI + role-based access.

Track the why, not just the what

"Updated qa.043 reference" is a useless commit message. "Updated qa.043: original reference was outdated; Tokyo population revised to 2024 figures per provenance link" is a useful one. Future-you, debugging a regression, will thank present-you for the trail.

Code

Rubric template — markdown, lives next to the dataset·markdown

# rubric.md

## Question
Does the assistant's answer correctly summarize the article in <= 100 words?

## Pass criteria (all must hold)
- Mentions the central claim of the article
- Mentions the strongest supporting evidence
- Word count <= 100
- No factual claims absent from the article

## Fail examples
- Summary contains a fact not in the article (hallucination)
- Word count exceeds 100
- Summary changes the central claim

## Edge cases
- If the article has multiple central claims, mentioning either passes
- A refusal ("I cannot summarize this") fails
- Stylistic choices (formal vs casual) do NOT influence the rating

## Calibration set
Label cases qa.001 through qa.020 first. Discuss disagreements before continuing.

Dataset version log entry·json

{
  "version": 7,
  "date": "2026-04-15",
  "author": "alice",
  "changes": [
    {"id": "qa.043", "action": "updated_reference", "reason": "Tokyo population revised to 2024 figures", "prev_value": "~13.5M", "new_value": "~14M city, ~37M metro"},
    {"id": "qa.108", "action": "added", "source": "production_log", "date_collected": "2026-04-12"},
    {"id": "qa.077", "action": "removed", "reason": "duplicate of qa.034"}
  ]
}

Annotation Guidelines and Versioning

The annotation rubric is the contract

What a good rubric contains

Versioning the dataset like code

Track the why, not just the what

Code

External links

Exercise

Progress

Comments 0