Data Collection & Synthetic Generation

Four practical sources

Manual curation — write examples by hand. Highest quality, most expensive in human time. Necessary for the first 20–50 gold examples.
Production logs — collect real user interactions (with consent and privacy review), filter for quality. The closest you get to real distribution.
Synthetic generation — use a strong model (GPT-4o, Claude 3.5 Sonnet) to generate examples seeded by your gold set. Scales fastest.
Distillation — feed real or synthetic inputs through a large teacher model and use its outputs as training data for a smaller student.

The synthetic-generation pipeline that doesn't drift

The mistake: ask a strong model to "generate 1,000 examples about X" and dump the result. The model collapses into a few templates and you train on 1,000 nearly-identical rows.

The fix: enumerate the axes of variation explicitly (topic, difficulty, tone, length, edge case), generate one example per cell of the axis grid, and review a 20% sample by hand.

Code

Grid-based synthetic generator that produces real diversity·python

import json, itertools
from openai import OpenAI

teacher = OpenAI()

TOPICS = ["refund request", "shipping delay", "product defect",
          "account locked", "billing question"]
DIFFICULTIES = ["clear request", "ambiguous", "angry", "polite-but-vague"]
LENGTHS = ["one sentence", "a short paragraph", "multiple paragraphs"]

SYSTEM = """You generate realistic customer-support training pairs.
Output JSON: {"user": "...", "assistant": "..."}.
The assistant should be empathetic, concise, and follow the company's
formal-but-warm voice."""

def generate_pair(topic: str, difficulty: str, length: str) -> dict:
    prompt = (f"Topic: {topic}. Customer mood: {difficulty}. "
              f"Customer message length: {length}. Generate one realistic pair.")
    r = teacher.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": prompt},
        ],
        response_format={"type": "json_object"},
        temperature=0.8,
    )
    pair = json.loads(r.choices[0].message.content)
    return {"messages": [
        {"role": "system", "content": "You are a customer support agent."},
        {"role": "user", "content": pair["user"]},
        {"role": "assistant", "content": pair["assistant"]},
    ]}

with open("synthetic.jsonl", "w") as f:
    for topic, diff, length in itertools.product(TOPICS, DIFFICULTIES, LENGTHS):
        ex = generate_pair(topic, diff, length)
        f.write(json.dumps(ex) + "\n")
print("Generated", len(TOPICS) * len(DIFFICULTIES) * len(LENGTHS), "examples")

Data Collection & Synthetic Generation

Four practical sources

The synthetic-generation pipeline that doesn't drift

Code

External links

Exercise

Progress

Comments 0