Storage Formats and Maintenance

JSONL is the eval lingua franca

Every eval framework worth using consumes JSONL — newline-delimited JSON, one record per line. It is grep-able, diff-friendly, streamable, and has zero schema overhead. Use it.

Schema discipline

Even though JSONL is schemaless, your records should not be. Pin a schema, validate it on write, and version it when it changes.

Required: id, input, tags.
Optional: reference, acceptable_alternatives, expected, metadata, source.
Auditing: created_at, created_by, version.

Where to store

Where	When
Repo (datasets/*.jsonl)	< 5MB total. Reviewed via PR. Easiest.
S3 / R2 / GCS	Larger. Versioned with object lifecycle.
Hugging Face Hub	Public datasets you want to share.
Braintrust / DeepEval cloud	UI annotation + versioning + eval runs in one place.
Argilla / Label Studio	Self-hosted UI for active annotation.

Principle: The dataset format that makes diffs readable is the dataset format you will actually maintain. JSONL beats Parquet for eval datasets you touch by hand.

Maintenance rituals

Quarterly review — sample 50 random cases. Are they still representative of production?

Stale flag — when production sampling reveals an emerging input cluster not in the dataset, add it within the same week. Slow refresh kills the eval suite.

Retirement — cases whose behavior is no longer relevant get an archived: true flag rather than deletion. You want the historical trail.

Code

JSONL schema validation·python

from pydantic import BaseModel, Field
from typing import Optional
import json

class EvalCase(BaseModel):
    id: str
    input: str
    reference: Optional[str] = None
    acceptable_alternatives: list[str] = Field(default_factory=list)
    tags: list[str] = Field(default_factory=list)
    source: str = "unknown"
    created_at: Optional[str] = None
    version: int = 1

def validate_jsonl(path):
    bad = []
    with open(path) as f:
        for i, line in enumerate(f, 1):
            try:
                EvalCase.model_validate_json(line)
            except Exception as e:
                bad.append((i, str(e)))
    return bad

errors = validate_jsonl("datasets/regression.jsonl")
if errors:
    raise SystemExit(f"{len(errors)} bad records: {errors[:3]}")

Pre-commit hook — block dataset PRs that fail validation·yaml

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: validate-eval-datasets
        name: Validate eval JSONL datasets
        entry: python scripts/validate_datasets.py
        language: system
        files: 'datasets/.*\.jsonl$'
        pass_filenames: true

Storage Formats and Maintenance

JSONL is the eval lingua franca

Schema discipline

Where to store

Maintenance rituals

Code

External links

Exercise

Progress

Comments 0