Confusion Matrix and Metrics

~28 min · metrics, confusion-matrix

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The four cells that explain everything

True positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Every classification metric is a ratio of these four numbers. If you internalize the four cells, you can derive precision, recall, F1, and accuracy on the whiteboard without looking them up.

Definitions worth memorizing

Precision = TP / (TP + FP) — of the ones we predicted positive, how many were right?
Recall = TP / (TP + FN) — of the actual positives, how many did we catch?
F1 = harmonic mean of precision and recall.
Accuracy = (TP + TN) / total — useful only when classes are balanced.

Read the matrix before you read the metric

Always print the confusion matrix and a per-class report before reporting any single number. The four cells often reveal that your "95% accuracy" model never predicts the minority class.

Code

Confusion matrix + classification report·python

from sklearn.metrics import confusion_matrix, classification_report

preds = (probs >= 0.5).astype(int)
cm = confusion_matrix(y_val, preds)
print(cm)
print(classification_report(y_val, preds, digits=3))

Pretty plot for stakeholders·python

from sklearn.metrics import ConfusionMatrixDisplay
import matplotlib.pyplot as plt

ConfusionMatrixDisplay.from_predictions(y_val, preds, normalize="true")
plt.title("Validation confusion matrix (row-normalized)")
plt.show()

External links

Exercise

For your classifier, plot the row-normalized confusion matrix. Identify the worst recall class. Sample 20 errors from that class and label them as "hard", "ambiguous", or "data bug". Decide whether to fix data, fix labels, or accept the limit.