Why Fine-Tune (vs. Prompt, vs. RAG)

Three knobs in the tool drawer

Before you fine-tune, ask whether you should. The decision tree:

Prompt engineering — adjust the system prompt, the chat template, the few-shot examples. Cost: hours. Solves: behavior steering, simple format requirements, persona, style.
RAG (retrieval-augmented generation) — pull relevant context from a vector store at inference time. Cost: days. Solves: domain knowledge, freshness, citation, large knowledge bases.
Fine-tuning — modify the weights with task-specific data. Cost: weeks (data + compute + eval). Solves: deep style/domain mastery, latency reduction (model knows X without re-explaining), vocabulary adaptation.

The fine-tune trigger conditions

Prompt-only solutions are unreliable on your eval set after multiple iterations.
RAG context windows are blowing your latency budget.
You have at least 1k high-quality examples (often 10k+ for visible quality jumps).
You have GPU budget for both training and ongoing eval.

If you're fine-tuning to "teach the model facts," that's almost always a RAG problem in disguise. Fine-tuning teaches behavior, not facts.

Code

Fine-tune vs prompt vs RAG: pseudo-decision·python

def choose_strategy(eval_score_with_prompt, latency_budget_ms, has_facts_to_inject, num_examples):
    if eval_score_with_prompt >= target_score:
        return "stay with prompt engineering"
    if has_facts_to_inject and latency_budget_ms > 500:
        return "RAG"
    if num_examples >= 1000 and gpu_budget_available:
        return "fine-tune (start with LoRA)"
    return "collect more data, retry"

Why Fine-Tune (vs. Prompt, vs. RAG)

Three knobs in the tool drawer

The fine-tune trigger conditions

Code

External links

Exercise

Progress

Comments 0