~24 min · synthetic, distillation, flywheel, ab-testing
Level 0Observer
0 XP0/43 lessons0/11 achievements
0/120 XP to next level120 XP to go0% complete
The distillation pipeline (production view)
The most common real-world pattern: large teacher generates training data, smaller student is fine-tuned on it. This is how teams cut the inference cost of a heavy GPT-4o pipeline by 10× while keeping ~95% of the quality.
The data flywheel
Deploy your fine-tuned model in production behind A/B test.
Collect real user interactions (with consent + privacy review).
Filter for quality — keep interactions rated positively or where the user accepted the response.
Fine-tune again on the combined original + new data.
Repeat — each cycle produces a better dataset and a better model.
Code
Diverse synthetic generation across an axis grid·python
import json, itertools
from openai import OpenAI
client = OpenAI()
def generate_pair(topic: str, difficulty: str) -> dict:
r = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "system",
"content": (
"Generate a training example for a Python coding assistant. "
"Output JSON: {\"question\": \"...\", \"answer\": \"...\"}. "
"Answer should include code AND a one-paragraph explanation. "
f"Difficulty: {difficulty}."
),
}, {
"role": "user",
"content": f"Topic: {topic}",
}],
response_format={"type": "json_object"},
temperature=0.8,
)
return json.loads(r.choices[0].message.content)
TOPICS = ["list comprehensions", "decorators", "async/await",
"file handling", "error handling", "classes", "generators"]
DIFFICULTIES = ["beginner", "intermediate", "advanced"]
examples = []
for topic, diff in itertools.product(TOPICS, DIFFICULTIES):
for _ in range(10): # 10 per cell
pair = generate_pair(topic, diff)
examples.append({"messages": [
{"role": "system", "content": "You are a Python expert."},
{"role": "user", "content": pair["question"]},
{"role": "assistant", "content": pair["answer"]},
]})
print(f"Generated {len(examples)} examples")
# 7 topics × 3 difficulties × 10 = 210 examples
Sketch a 6-month data flywheel plan for one of your AI features. Document: what you'd log, what consent flow you'd add, what quality filter, what retraining cadence (monthly?), and what eval prevents regressions. This is the doc that would unlock continuous model improvement.
Progress
Progress is local-only — sign in to sync across devices.