Confusion Matrix and Metrics

Every classification result enters one of four cells

A binary decision is a true positive, false positive, true negative, or false negative. Define the positive class before using abbreviations. In one product “positive” means disease found; in another it means a transaction blocked. Reversing that choice reverses the business interpretation.

Translate the cells into real events

A false positive may be a legitimate customer blocked or an hour of wasted review. A false negative may be missed fraud or an untreated patient. Report raw counts in those terms so cost and accountability remain visible.

Precision asks whether an alert can be trusted

Precision is TP / (TP + FP): among predicted positives, how many were truly positive? It matters when review capacity is limited or unnecessary action is costly. Raising the threshold can inflate precision by producing very few alerts, so report recall and alert volume alongside it.

Recall asks how many positives were found

Recall is TP / (TP + FN): among actual positives, how many did the model catch? It matters when missing disease or major fraud is dangerous. Calling everything positive yields recall of one, so include precision and false-positive burden.

F1 and accuracy carry assumptions

F1 is the harmonic mean of precision and recall; it ignores true negatives and treats their balance symmetrically. Accuracy counts all correct decisions and is intuitive only when prevalence and error costs make it meaningful. With one-percent prevalence, a model that never acts can be 99 percent accurate.

Normalize in the direction of the question

Row-normalizing by actual class shows per-class recall. Column-normalizing by predicted class shows per-class precision. Always retain raw counts because 90 percent from ten examples does not carry the certainty of 90 percent from ten thousand.

Use the matrix as the entrance to error analysis

Print the confusion matrix and class report before relying on one summary metric, and recompute them at the intended operating threshold. Sample examples from the costliest cell and tag hard cases, ambiguous labels, data errors, missing features, shift, and genuine model limitations. The matrix becomes useful when each failure pattern points to a different repair.

Code

Confusion matrix + classification report·python

from sklearn.metrics import confusion_matrix, classification_report

preds = (probs >= 0.5).astype(int)
cm = confusion_matrix(y_val, preds)
print(cm)
print(classification_report(y_val, preds, digits=3))

Pretty plot for stakeholders·python

from sklearn.metrics import ConfusionMatrixDisplay
import matplotlib.pyplot as plt

ConfusionMatrixDisplay.from_predictions(y_val, preds, normalize="true")
plt.title("Validation confusion matrix (row-normalized)")
plt.show()