When Fine-Tuning Loses

The four anti-patterns

The mirror of the previous lesson. These are the situations where fine-tuning is the wrong tool, and where most failed projects start.

1. One-off tasks

If you only need a behavior occasionally, prompt engineering is faster, cheaper, and more flexible. Fine-tuning is for repeated, consistent behavior at production frequency.

2. Rapidly changing knowledge

Fine-tuning bakes facts into weights at training time. If your data changes weekly (news, prices, product catalogs, ticket queues), use RAG. The model cannot "unlearn" outdated facts without retraining, and you do not want to be retraining every Tuesday.

3. Tiny datasets (< 50 examples)

Most fine-tuning setups need at least 50–100 examples to see improvement, and 500–1,000 for reliable results. Below 50 examples you usually get one of two failures: no measurable change, or aggressive overfitting that destroys generality.

4. Adding factual knowledge

The most common misconception. Fine-tuning does not reliably teach a model new facts. It teaches behaviors — how to format, when to refuse, what tone to use. For knowledge, use RAG. Trying to teach facts via fine-tuning is the single most common project-failure pattern.

Catastrophic forgetting

The shadow risk that ties them all together: aggressive full fine-tuning can overwrite the model's general capabilities. A model fine-tuned heavily on legal text can become measurably worse at creative writing or basic reasoning. This is one of the deepest reasons parameter-efficient methods like LoRA exist (Track 4) — by updating only a tiny fraction of parameters, they minimize the risk of stomping on the rest of the model.

Code

Catastrophic-forgetting regression check·python

from openai import OpenAI

client = OpenAI()
GENERAL_TESTS = [
    "Summarize the plot of Pride and Prejudice in 3 sentences.",
    "Write a Python function that returns the Nth Fibonacci number.",
    "What is the difference between TCP and UDP?",
]

def regression_check(model_id: str) -> None:
    for q in GENERAL_TESTS:
        r = client.chat.completions.create(
            model=model_id,
            messages=[{"role": "user", "content": q}],
            temperature=0,
        )
        print(f"=== {q[:40]}\n{r.choices[0].message.content}\n")

# Run against the BASE model AND the FINE-TUNED model.
# If the fine-tuned model is visibly worse on general tasks,
# you've forgotten too much.
regression_check("gpt-4.1-mini-2025-04-14")
regression_check("ft:gpt-4.1-mini-2025-04-14:org:custom:abc123")

Exercise

Take a fine-tuned model your team has trained (or a public one from the Hugging Face Hub). Write a 10-question general-capability regression suite — math, coding, summarization, common-sense reasoning, simple instructions. Run it on the base model and the fine-tuned model. Document any drops you find.