Perplexity, F1, and Classification Metrics

Pick the metric that matches the task shape

Perplexity, accuracy, precision, recall, F1, AUC — these are the bread-and-butter metrics of classical ML. They still apply when your LLM output reduces to a class label or a probability.

Perplexity

Perplexity measures how surprised a language model is by a piece of text. Lower is better. It is mostly relevant for evaluating base language models (does this model assign reasonable probability to natural English?), not for evaluating product features. Useful as a smoke test that fine-tuning did not destroy the model.

Classification metrics

For tasks like intent classification, content moderation, sentiment analysis, or any "label this output" task, the standard four apply:

Accuracy — fraction correct overall. Useful only when classes are balanced.
Precision — of items predicted positive, how many actually are. "Did I cry wolf?"
Recall — of actual positives, how many did I find. "Did I miss any?"
F1 — harmonic mean of precision and recall. The default reporting metric for imbalanced binary classification.

Principle: When your task has a label, report precision and recall separately. F1 alone hides which side is bleeding.

Multi-class and multi-label

For multi-class problems use macro-F1 (unweighted average across classes — treats classes equally) or weighted-F1 (weighted by support — reflects class frequency). Macro punishes you for poor performance on rare classes; weighted is closer to user-perceived accuracy.

For multi-label (an output may belong to multiple classes), F1 is computed per label then averaged.

Confusion matrix is the diagnostic tool

A single F1 number is the headline; the confusion matrix is the article. It shows you which classes are getting confused for which other classes. Always log the confusion matrix when reporting classification metrics.

Code

Sklearn classification report·python

from sklearn.metrics import classification_report, confusion_matrix

y_true = ["refund", "complaint", "praise", "refund", "complaint", "praise", "refund"]
y_pred = ["refund", "complaint", "praise", "complaint", "complaint", "praise", "refund"]

print(classification_report(y_true, y_pred, digits=3))
# Per-class precision, recall, F1, support, then macro & weighted averages.

print(confusion_matrix(y_true, y_pred, labels=["refund", "complaint", "praise"]))
# Rows = true class, columns = predicted class. Off-diagonal = errors.

Perplexity in practice — base model smoke test·python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "meta-llama/Llama-3.2-1B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

text = "The quick brown fox jumps over the lazy dog."
ids = tok.encode(text, return_tensors="pt")
with torch.no_grad():
    loss = model(ids, labels=ids).loss
    perplexity = torch.exp(loss).item()
print(f"perplexity: {perplexity:.2f}")
# A reasonable English sentence on a 1B model: ~10-30.
# Above ~100 means something is broken.

Perplexity, F1, and Classification Metrics

Pick the metric that matches the task shape

Perplexity

Classification metrics

Multi-class and multi-label

Confusion matrix is the diagnostic tool

Code

External links

Exercise

Progress

Comments 0