Class Imbalance

The accuracy trap

If 1% of users churn, a model that predicts "no one churns" is 99% accurate and 0% useful. Class imbalance first exposes a problem with the metric, not necessarily the model. Measure how many rare positives you recover and what false alerts cost instead of celebrating the majority class.

Read the costs from the confusion matrix

Translate false positives and false negatives into actions, money, or review time. Missing fraud and reviewing a legitimate transaction do not have equal consequences. Choose PR-AUC, F-beta, recall at a fixed precision, or expected cost from that operational asymmetry.

Three honest responses

Change the metric — PR-AUC, F-beta, recall at fixed precision, expected cost.
Change the threshold — leave the model as-is and pick an operating point that respects cost and team capacity.
Change the sampling — compare class weights, majority downsampling, and SMOTE for tabular minority synthesis.

PR-AUC is often more revealing

ROC-AUC can remain high when a huge negative class makes the false-positive rate look small. Precision-recall curves expose how pure the alerts are as recall rises. Report PR-AUC beside the positive prevalence and inspect the exact operating point the team can use.

Resample only inside training folds

Do not oversample before the train/validation split. Synthetic or duplicated minority examples can leak near-copies into validation. Put resampling inside an imbalanced-learn pipeline so each cross-validation fold changes only its training portion while validation keeps the real class prevalence.

Fifty-fifty is not the objective

Start with the nearly free class_weight="balanced" baseline. Aggressive resampling can overfit and distort probability calibration. Evaluate every method on data with the production prevalence, recalibrate if necessary, and choose the one that minimizes the metric and cost you actually care about.

Code

Use class_weight='balanced' before reaching for SMOTE·python

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score

model = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X_tr, y_tr)
pr_auc = average_precision_score(y_val, model.predict_proba(X_val)[:, 1])

SMOTE inside a sklearn-imblearn pipeline·python

from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline
from sklearn.ensemble import GradientBoostingClassifier

pipe = Pipeline([
    ("smote", SMOTE(random_state=7)),
    ("clf", GradientBoostingClassifier()),
])

The accuracy trap

Read the costs from the confusion matrix

Three honest responses

PR-AUC is often more revealing

Resample only inside training folds

Fifty-fifty is not the objective

Code

External links

Exercise

Progress

Comments 4