Fine-Tuning vs Prompting vs Feature Extraction

Three adaptation strategies, ordered by cost

Prompting — describe the task in natural language. Zero training cost, fast iteration, works surprisingly well for many tasks. Limited by prompt context length and prompt-engineering ceiling.
Feature extraction — embed your data with a pretrained encoder, train a small classical head (linear, MLP, gradient boosting) on the embeddings. No backbone training. Cheap, scalable.
Fine-tuning — actually update the backbone (full or LoRA) on your task. Highest ceiling, biggest engineering investment, most overfitting risk.

Tip: Try them in this order and stop at the first one that meets your accuracy bar. Most production teams discover that prompting + a few examples handles the long tail; fine-tuning is reserved for the cases that genuinely need it.

When each wins

Prompting wins when the task is in the model's training distribution (general reasoning, common writing tasks, code completion in popular languages).
Feature extraction wins when you need fast inference at scale and the relevant signal is already captured in pretrained embeddings.
Fine-tuning wins when you need the model to behave consistently in a specific style, domain, or format — and you have hundreds to thousands of labeled examples.

The hybrid that often beats all three

RAG (prompting + retrieved context) + few-shot examples + a small reranker on top often beats fine-tuning for knowledge-intensive tasks. The infrastructure is more complex but the per-task adaptation cost stays low.

Principle: Adaptation cost matters as much as accuracy. A frozen-backbone classifier you can deploy on CPU often beats a fully fine-tuned 7B model that needs a GPU per request. Engineer the whole pipeline, not just the model.

Code

Feature extraction with a sentence-transformer·python

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

# 1. Frozen pretrained encoder
encoder = SentenceTransformer("all-MiniLM-L6-v2")

# 2. Embed your text
X_train_emb = encoder.encode(X_train_texts, normalize_embeddings=True)
X_val_emb   = encoder.encode(X_val_texts,   normalize_embeddings=True)

# 3. Cheap classical head
clf = LogisticRegression(max_iter=1000).fit(X_train_emb, y_train)
preds = clf.predict(X_val_emb)
print(f"f1: {f1_score(y_val, preds, average='macro'):.3f}")
# This baseline often beats hopeful fine-tuning attempts on small datasets.

Fine-Tuning vs Prompting vs Feature Extraction

Three adaptation strategies, ordered by cost

When each wins

The hybrid that often beats all three

Code

External links

Exercise

Progress

Comments 0