Preparing Training Data — JSONL, Templates, and Cleaning

Bad data beats small models on the loss curve, every time

The single biggest predictor of fine-tuning quality is the training data, not the rank, not the optimizer, not the number of iterations. A 7B model with 200 cleaned, well-formatted examples will outperform the same model with 2000 messy examples on most downstream tasks. Spend more time on the data than on hyperparameter tuning.

The format mlx-lm expects

mlx-lm wants JSONL — one JSON object per line. Two file naming conventions live in the data directory you point --data at:

train.jsonl — required. The training examples.
valid.jsonl — required. Validation examples used to compute val loss every --steps-per-eval iters.
test.jsonl — optional. Used by --test to evaluate after training.

Inside each .jsonl line, mlx-lm accepts two shapes: chat format (a messages array, just like OpenAI's chat-completions) or completion format (a prompt + completion pair, for older non-chat training).

The chat format — the right default

Each line is a JSON object with a messages field. Each message has role (system / user / assistant) and content. mlx-lm will apply the model's chat template at training time, so the formatting matches what the model will see at inference. This is non-negotiable for chat-tuned base models — train on the chat-templated form or the resulting fine-tune behaves badly at inference.

The cleaning step everyone skips

Before training on any dataset (your own or one you pulled from HF), do these passes:

Deduplicate. Identical or near-identical examples bias the model and inflate apparent training-set size.
Length-filter. Drop examples that are absurdly short (no signal) or longer than your --max-seq-length (would be truncated, often mid-thought).
Quality-filter. Even a quick eyeball of the bottom 5% of examples by your judgment catches a lot — typos, broken formatting, off-topic noise.
Split without leakage. Hold out validation and test sets from the same dataset; make sure they're disjoint from training. Random split is fine for most cases; for very small datasets, stratify by your task's labels.

If you want a real-world dataset to play with

The mlx-community org on Hugging Face hosts a few small datasets that work out of the box with mlx-lm — mlx-community/wikisql (SQL generation, classic) is the canonical demo set. For your own task, even 100 well-curated examples are a respectable starting point.

Code

Chat-format JSONL — the canonical training data shape·python

# Each line is one example with a 'messages' array.
import json

examples = [
    {
        "messages": [
            {"role": "system",    "content": "You are a SQL expert."},
            {"role": "user",      "content": "How many users joined in 2024?"},
            {"role": "assistant", "content": "SELECT COUNT(*) FROM users WHERE YEAR(joined_at) = 2024;"},
        ]
    },
    {
        "messages": [
            {"role": "system",    "content": "You are a SQL expert."},
            {"role": "user",      "content": "List all products under $10."},
            {"role": "assistant", "content": "SELECT * FROM products WHERE price < 10;"},
        ]
    },
]

with open("./my-data/train.jsonl", "w") as f:
    for ex in examples:
        f.write(json.dumps(ex) + "\n")

# Then point mlx-lm at the directory:
#   python -m mlx_lm lora --model ... --train --data ./my-data

Quick deduplication + length filter·python

import json, hashlib

def line_hash(ex):
    # Hash the messages content for dedup
    return hashlib.md5(json.dumps(ex["messages"], sort_keys=True).encode()).hexdigest()

def total_chars(ex):
    return sum(len(m["content"]) for m in ex["messages"])

seen = set()
kept, dropped = [], 0
for line in open("./my-data/train.jsonl"):
    ex = json.loads(line)
    h = line_hash(ex)
    n = total_chars(ex)
    if h in seen or n < 50 or n > 8000:   # adjust thresholds for your task
        dropped += 1
        continue
    seen.add(h)
    kept.append(ex)

print(f"kept {len(kept)}, dropped {dropped}")
with open("./my-data/train.cleaned.jsonl", "w") as f:
    for ex in kept:
        f.write(json.dumps(ex) + "\n")

Exercise

Pick a tiny domain you know well — could be SQL queries, regex patterns, witty haikus, summarizing tech RSS, anything narrow. Hand-write 30 chat-format JSONL examples (system + user + assistant). Save as train.jsonl, hold 5 out as valid.jsonl. Run the dedup + length filter from the second code block. The exercise is to feel the work — most of it is curation, not training.