ROC-AUC vs PR-AUC

Move the threshold to see the tradeoff

A high threshold predicts few positives; lowering it catches more positives and usually creates more false alerts. ROC and precision-recall curves trace that exchange across possible thresholds. Both use the continuous score rather than the final label, and neither chooses the operating point for you.

What ROC-AUC measures

The ROC curve plots true-positive rate against false-positive rate. ROC-AUC can be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative. One is perfect and 0.5 is random ordering. It is convenient for balanced classes and broad ranking comparison.

Why rare positives can fool ROC

One percent false positives among one million negatives is a low rate but ten thousand alerts. With a huge negative class, ROC-AUC may look strong while operational precision is unusable. Always translate the rate back into counts.

What PR-AUC measures

The precision-recall curve shows how pure predicted positives remain as the model recovers more actual positives. PR-AUC or average precision therefore exposes alert burden directly in fraud, churn, and anomaly detection, where the positive class is rare.

The baseline depends on prevalence

ROC-AUC has a random baseline of 0.5. PR-AUC's random baseline is the positive prevalence: if one percent of users churn, it is 0.01. A PR-AUC of 0.30 is thirty times random in that setting. Report the prevalence and a simple baseline beside the score.

AUC does not choose deployment

An area averages thresholds the product may never use. A model with better PR-AUC can still be worse at precision 0.7 or a top-200 capacity. Zoom into the operating region and compare precision, recall, count, calibration, and expected cost there.

Re-evaluate when prevalence changes

Precision and PR-AUC change when the positive rate changes, even if ranking is similar. Evaluate on data with production prevalence after any resampling and monitor prevalence over time. This separates a changed environment from a changed ranking model.

Code

Both AUCs side by side·python

from sklearn.metrics import roc_auc_score, average_precision_score

roc = roc_auc_score(y_val, probs)
pr = average_precision_score(y_val, probs)
prevalence = y_val.mean()
print(f"ROC-AUC={roc:.3f}  PR-AUC={pr:.3f}  prevalence={prevalence:.3f}")

Plot the PR curve to see where the model breaks·python

from sklearn.metrics import PrecisionRecallDisplay
import matplotlib.pyplot as plt

PrecisionRecallDisplay.from_predictions(y_val, probs)
plt.title("Validation precision-recall curve")
plt.grid(True)
plt.show()