When Fine-Tuning Wins

The five honest sweet spots

Fine-tuning earns its keep when prompt engineering and RAG hit a ceiling you can describe. These five categories are where it consistently pays back the effort.

1. Consistent style and format

JSON with exact fields, a particular brand voice, a structured-report template — fine-tuning takes a 90% prompt-only result to ~99–100% format compliance. The last 10 points matter when a downstream system parses the output.

2. Domain jargon and terminology

Medical, legal, financial, telecom, manufacturing — fine-tuning teaches the model your domain's vocabulary so it speaks fluently instead of awkwardly.

3. Reliable structured output

Function calls, API parameters, SQL queries, tool-use payloads. Fine-tuning dramatically raises the rate at which the model produces parseable output on the first try, which compounds inside an agent loop.

4. Latency and cost (distillation)

The most economically important pattern: a strong model (GPT-4o, Claude, Llama 3.1 70B) generates high-quality outputs, those become training data, and a smaller model (GPT-4.1-mini, Llama 3.1 8B) is fine-tuned to match ~95% of the quality at ~10% of the cost. This is model distillation.

5. Reducing prompt-token cost

If your system prompt is 1,500 tokens of carefully tuned instructions, you pay for those 1,500 tokens on every request. Fine-tuning bakes the instructions into the weights so you can ship a 50-token system prompt instead. At scale, this dominates inference cost.

The pattern check

If your use case fits two or more of the five, fine-tuning is almost certainly worth pursuing. If it fits zero, it almost certainly is not.

Code

Distillation pipeline in 25 lines·python

import json
from openai import OpenAI

teacher = OpenAI()
SYSTEM = "You are a senior technical writer producing concise, accurate answers."

def teach(question: str) -> dict:
    r = teacher.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": question},
        ],
        temperature=0.3,
    )
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": question},
        {"role": "assistant", "content": r.choices[0].message.content},
    ]}

questions = [line.strip() for line in open("questions.txt") if line.strip()]
with open("distill.jsonl", "w") as f:
    for q in questions:
        f.write(json.dumps(teach(q)) + "\n")
# Now fine-tune gpt-4.1-mini on distill.jsonl. Same vibe, ~10% of the cost.

Exercise

List the top 3 LLM API calls in your stack by monthly token cost. For each, score the five sweet spots (0–2 each). Pick the highest-scoring one and write a one-page distillation brief: teacher model, target student model, expected cost reduction, expected quality drop, smallest dataset that would prove the case.