Common Failures & Debugging

The top-10 fine-tuning failures (and fixes)

#	Symptom	Likely cause	Fix
1	Model outputs gibberish	Wrong chat template or tokenizer mismatch	Use the matching tokenizer; verify chat template at training and inference are identical.
2	No measurable improvement	Too few examples or low data quality	500+ examples; review every one of the first 50.
3	Validation loss rises early	Too many epochs, dataset too small	Reduce epochs, increase dropout, add diverse data.
4	Lost general ability (forgetting)	Catastrophic forgetting from full FT	Use LoRA instead; lower learning rate.
5	Inconsistent format in outputs	Training data has inconsistent format	Standardize all examples to one format before training.
6	Loss doesn't decrease at all	Learning rate too low, or data format wrong	Increase LR; verify data is reaching the model unchanged.
7	OOM (out of memory)	Model or batch too large for GPU	Use QLoRA; reduce batch; enable gradient checkpointing.
8	Slow training	No Flash Attention, no bf16	Enable Flash Attention 2 + bf16.
9	Model hallucinates more after FT	Training data contains errors or hallucinations	Audit data; remove anything you can't verify.
10	Works in eval, fails in production	Eval data too similar to training	Build a real production-derived test set; never reuse training distribution for eval.

Code

Three-step debug protocol when something looks wrong·python

# 1. Manually inspect 5 random training examples
import json, random
with open("train.jsonl") as f:
    examples = [json.loads(line) for line in f]
for ex in random.sample(examples, 5):
    for m in ex["messages"]:
        print(f"[{m['role']}] {m['content'][:240]}")
    print("---")

# 2. Verify the tokenized output looks correct
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
for ex in random.sample(examples, 1):
    formatted = tok.apply_chat_template(ex["messages"], tokenize=False)
    print("=== formatted ===")
    print(formatted[:1000])
    tokens = tok(formatted)["input_ids"]
    print(f"=== token count: {len(tokens)} ===")

# 3. Run a tiny training run (10 examples, 1 epoch) to verify pipeline
# If THAT crashes, the pipeline is broken, not the data scale.

The top-10 fine-tuning failures (and fixes)

Code

External links

Exercise

Progress

Comments 0