Threshold Tuning and Business Cost

~30 min · threshold, cost-matrix

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

0.5 is almost never the right threshold

Default 0.5 is a convention, not a recommendation. The right threshold depends on the cost of false positives, the cost of false negatives, and the operating capacity of the team. Move the threshold to match the business, then defend the choice with a number, not a vibe.

Three honest threshold strategies

Cost-minimizing — set the threshold where expected cost is minimum.
Recall at precision floor — pick the threshold where precision ≥ P (often used when team capacity is fixed).
Top-K capacity — score everyone, take the top K, and let downstream act on those K (always a thresholded ranking under the hood).

Logging and monitoring the threshold

Once you ship a threshold, log it next to the model version. Drift can change the right threshold even when the model is unchanged; monitor the actual operating point against the intended one weekly.

Code

Pick the threshold that minimizes expected cost·python

import numpy as np

thresholds = np.linspace(0.01, 0.99, 99)
cost_fp, cost_fn = 5, 50
best_t, best_cost = 0.5, float("inf")
for t in thresholds:
    preds = (probs >= t).astype(int)
    fp = ((preds == 1) & (y_val == 0)).sum()
    fn = ((preds == 0) & (y_val == 1)).sum()
    cost = fp * cost_fp + fn * cost_fn
    if cost < best_cost:
        best_cost, best_t = cost, t
print(f"chosen threshold {best_t:.2f}  expected cost {best_cost}")

Recall at fixed precision·python

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_val, probs)
ok = precision[:-1] >= 0.70
if ok.any():
    idx = np.argmax(recall[:-1][ok])
    chosen = thresholds[ok][idx]
    print(f"threshold for precision≥0.70: {chosen:.3f} → recall {recall[:-1][ok][idx]:.3f}")

External links

Exercise

For your binary classifier, write the cost matrix in concrete units. Compute the cost-minimizing threshold and the recall-at-precision-0.7 threshold. Pick one as the operating point and document why.