Data Quality Beats Data Quantity

The single biggest lever

If you remember one thing from this entire quest, remember this: data quality dominates everything else. Hyperparameters move the needle by single-digit percent. Data quality moves it by tens.

What "quality" actually means

Correctness — every assistant response is factually accurate and well reasoned. Wrong examples teach wrong behavior with terrifying fidelity.
Consistency — similar inputs get similar-shaped outputs. Inconsistent format teaches the model that any format is fine.
Diversity — cover the real range of inputs. 500 happy-path examples means your fine-tuned model collapses on the 501st edge case.
Completeness — each example fully demonstrates the desired behavior, including any "thinking out loud."
Natural language — examples sound like real interactions, not synthetic templates.

The curation workflow that survives

Write a one-page spec: what behavior, in what tone, in what format?
Hand-write 20–50 "gold standard" examples from the spec. Slow, deliberate, edited.
Optionally generate more examples with a strong teacher model using gold examples as few-shot anchors.
Review every generated example. Reject or rewrite the bad ones. No exceptions.
Test a tiny training run (10–50 examples) end-to-end before scaling up.
Iterate: if the model keeps failing on certain patterns, add more examples of those — don't just throw more random data.

The shortcut to refuse

Scraping forums, dumping unfiltered LLM outputs, recycling production logs without review — every shortcut here gets faithfully learned. The cost of low-quality data is paid in inference time, when the model behaves exactly as you trained it.

Code

An 'is this example actually good?' review checklist as code·python

import json

REJECTIONS = []

def review(example: dict) -> str | None:
    msgs = example["messages"]
    user = next(m for m in msgs if m["role"] == "user")["content"]
    assistant = next(m for m in msgs if m["role"] == "assistant")["content"]

    if len(user.strip()) < 5:
        return "user input too short"
    if len(assistant.strip()) < 10:
        return "assistant response too short"
    if assistant.strip().lower().startswith("i'm sorry") and "refuse" not in user.lower():
        return "unwarranted refusal"
    if assistant.count("```") % 2 != 0:
        return "unmatched code fence"
    return None

with open("raw.jsonl") as f, open("clean.jsonl", "w") as out:
    kept = 0
    for line in f:
        ex = json.loads(line)
        problem = review(ex)
        if problem:
            REJECTIONS.append((problem, ex))
        else:
            out.write(line)
            kept += 1

print(f"Kept {kept}, rejected {len(REJECTIONS)}")

Exercise

Take 20 examples from any candidate fine-tuning dataset and run them through a printed 5-criterion checklist (correctness, consistency, diversity, completeness, natural language). Score each example 0–2 on each criterion. Reject anything below 7/10. How many of your 20 survive? That is your real dataset size.