Preparing & Uploading Data

~22 min · openai, jsonl, upload, validation

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

OpenAI's data requirements

OpenAI requires JSONL with the chat-messages format. Minimum 10 examples (50–100+ recommended), maximum file size 1 GB. System messages should be consistent across examples or omitted entirely.

The pre-upload validator

Validate your file before uploading. The OpenAI API will reject malformed files, but local validation gives you faster, more readable error messages.

Code

Validate locally, then upload·python

from openai import OpenAI
import json

client = OpenAI()

def validate_openai_data(filepath: str) -> int:
    """Returns number of valid examples; raises on hard errors."""
    with open(filepath) as f:
        data = [json.loads(line) for line in f]
    print(f"Number of examples: {len(data)}")
    if len(data) < 10:
        raise ValueError(f"Too few examples: {len(data)} < 10")

    for i, ex in enumerate(data):
        msgs = ex.get("messages", [])
        if not msgs:
            raise ValueError(f"Example {i}: missing 'messages'")
        if msgs[-1]["role"] != "assistant":
            raise ValueError(f"Example {i}: last message must be assistant")
        for j, m in enumerate(msgs):
            if not (m.get("content") or "").strip():
                raise ValueError(f"Example {i} msg {j}: empty content")

    chars = sum(len(json.dumps(ex)) for ex in data)
    tokens = chars // 4
    print(f"Estimated tokens: ~{tokens:,}")
    print(f"Estimated training cost on gpt-4.1-mini @ 3 epochs: "
          f"~${tokens * 3 / 1_000_000 * 0.80:.2f}")
    return len(data)

validate_openai_data("training_data.jsonl")

# Upload after local validation passes
training_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune",
)
print(f"File ID: {training_file.id}")

# Optional but recommended: validation file for monitoring
validation_file = client.files.create(
    file=open("validation_data.jsonl", "rb"),
    purpose="fine-tune",
)
print(f"Validation File ID: {validation_file.id}")

External links

Exercise

Prepare a 50-example training file + 10-example validation file in OpenAI chat format. Run the validator above. Upload both via the Files API. Record the file IDs — you'll use them in the next lesson.