Prompting vs RAG vs Fine-Tuning: When to Use Which

Three strategies for adapting a base model to your task. They're not mutually exclusive — most production systems use a mix. The question is which to start with.

Prompting (zero-shot / few-shot / in-context learning)

No model changes. Engineer better prompts, include examples, set the system message, structure the input. Fastest, cheapest, most flexible. Best for: prototyping, general tasks, tasks where the model already 'knows' enough but needs steering.

Retrieval-Augmented Generation (RAG)

Retrieve relevant documents at query time and include them in the prompt. No model changes. Best for: knowledge-intensive tasks, domains where information changes (so fine-tuning would go stale), tasks needing citations or grounding, tasks where the relevant data doesn't fit in the model's pretraining distribution.

Fine-tuning (full or LoRA / QLoRA)

Update model weights on your specific data. LoRA (Low-Rank Adaptation) updates only small adapter matrices, requiring far less compute than full fine-tuning. Best for: consistent style/format the model can't pick up from prompts, tasks where prompt engineering plus RAG isn't enough, distilling a large model's behavior into a smaller one.

Decision tree

Try prompting first. Usually clears 80% of needs.
If prompting fails because the model lacks specific knowledge, try RAG.
If both fail because you need consistent style/format or specialized behavior, then fine-tune. Fine-tuning is expensive, creates maintenance burden (must retrain for each base model update), and can reduce general capability.

Code

LoRA fine-tuning skeleton (PEFT)·python

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    torch_dtype=torch.bfloat16, device_map="auto",
)
config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj"],
    lora_dropout=0.05, task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
# Now only the LoRA adapter (~30M params for Llama 3 8B) is trainable.
# Train normally with HuggingFace Trainer or your own loop.
# Save the adapter, swap in/out without retraining the full model.

Prompting vs RAG vs Fine-Tuning: When to Use Which

Prompting (zero-shot / few-shot / in-context learning)

Retrieval-Augmented Generation (RAG)

Fine-tuning (full or LoRA / QLoRA)

Decision tree

Code

External links

Exercise

Progress

Comments 0