Threshold Tuning and Business Cost

0.5 is almost never the right threshold

Default 0.5 is a convention, not a recommendation. The right threshold depends on false-positive cost, false-negative cost, positive prevalence, and team capacity. Select it after model training and defend it with an operating calculation rather than a vibe.

Write the cost matrix in real units

Estimate review time and customer harm for a false positive, loss for a false negative, and the cost and benefit of acting on a true positive. Multiply each threshold's confusion-matrix counts by those values. When costs are uncertain, test optimistic, central, and pessimistic scenarios.

Three honest threshold strategies

Cost-minimizing chooses the threshold with minimum expected cost on representative validation data.
Recall at a precision floor maximizes recall subject to a requirement such as precision at least 0.7.
Top-K capacity scores everyone and sends only the highest K cases when downstream can process a fixed count.

Check sensitivity, not just the winner

If cost is nearly flat across a threshold range, prefer the simpler, more stable value. A sharp optimum may be validation noise. Report confidence intervals or fold variability and leave margin when a precision floor is mandatory.

Separate calibration data from final test data

Choose calibration and threshold settings without touching the frozen test set. Use the real production class prevalence, not a resampled training ratio. After every decision is fixed, evaluate expected cost and alert count once on test.

Logging and monitoring the threshold

Log the raw score, threshold, model version, policy version, and final action. Prevalence, costs, and capacity can move the best threshold even when model ranking is unchanged. Compare actual precision, recall, and alert volume with the intended operating point.

Change it as a product policy

Threshold updates need an owner, approval trail, staged rollout, and rollback just like model updates. A silent constant change can alter thousands of decisions without changing the model artifact, so policy history must remain independently auditable.

Code

Pick the threshold that minimizes expected cost·python

import numpy as np

thresholds = np.linspace(0.01, 0.99, 99)
cost_fp, cost_fn = 5, 50
best_t, best_cost = 0.5, float("inf")
for t in thresholds:
    preds = (probs >= t).astype(int)
    fp = ((preds == 1) & (y_val == 0)).sum()
    fn = ((preds == 0) & (y_val == 1)).sum()
    cost = fp * cost_fp + fn * cost_fn
    if cost < best_cost:
        best_cost, best_t = cost, t
print(f"chosen threshold {best_t:.2f}  expected cost {best_cost}")

Recall at fixed precision·python

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_val, probs)
ok = precision[:-1] >= 0.70
if ok.any():
    idx = np.argmax(recall[:-1][ok])
    chosen = thresholds[ok][idx]
    print(f"threshold for precision≥0.70: {chosen:.3f} → recall {recall[:-1][ok][idx]:.3f}")