Data Cleaning, Validation & Splits

The cleaning checklist

Deduplication — exact and near-duplicates. Near-dupe detection (MinHash, cosine on embeddings) catches what string-equality misses.
Format consistency — every example follows the same schema. Mix two formats and your model learns both badly.
Length filtering — drop too-short (trivial) and too-long (noisy or overflowing) examples.
Quality filtering — remove grammatical errors, hallucinated content, broken JSON, unmatched code fences.
Tokenization check — count tokens with the actual tokenizer, not character heuristics. Verify everything fits in the training context window.

The three-way split

Split	Share	Purpose
Train	~80%	What the model actually learns from.
Validation	~10%	Monitored during training. Detects overfitting; informs early stopping.
Test	~10%	Held out completely until after training. Final unbiased number.

Why validation is the canary

If validation loss starts rising while training loss keeps falling, the model is memorizing rows instead of learning behavior. That divergence is the overfitting signal — the moment to stop training, not when "the loss looks low."

Code

Defensive validator + 80/10/10 split for chat-format JSONL·python

import json, hashlib, random

def validate(filepath: str) -> tuple[list, list]:
    examples, errors, seen = [], [], set()
    with open(filepath) as f:
        for i, line in enumerate(f, 1):
            try:
                ex = json.loads(line)
            except json.JSONDecodeError as e:
                errors.append(f"line {i}: invalid JSON ({e})"); continue
            msgs = ex.get("messages")
            if not msgs:
                errors.append(f"line {i}: missing 'messages'"); continue
            if msgs[-1].get("role") != "assistant":
                errors.append(f"line {i}: last message must be assistant"); continue
            for j, m in enumerate(msgs):
                if not (m.get("content") or "").strip():
                    errors.append(f"line {i} msg {j}: empty content")
            h = hashlib.sha1(line.encode()).hexdigest()
            if h in seen:
                errors.append(f"line {i}: exact duplicate"); continue
            seen.add(h); examples.append(ex)
    return examples, errors

ex, errs = validate("clean.jsonl")
print(f"kept: {len(ex)}, errors: {len(errs)}")
random.Random(42).shuffle(ex)
n = len(ex)
splits = {"train": ex[:int(n*0.8)],
          "val":   ex[int(n*0.8):int(n*0.9)],
          "test":  ex[int(n*0.9):]}
for name, rows in splits.items():
    with open(f"{name}.jsonl", "w") as f:
        for r in rows: f.write(json.dumps(r) + "\n")
    print(f"{name}: {len(rows)}")

Data Cleaning, Validation & Splits

The cleaning checklist

The three-way split

Why validation is the canary

Code

External links

Exercise

Progress

Comments 0