Classification Basics

Probabilities first, labels second

A classifier usually produces a continuous score. A downstream policy turns that score into a label or action by applying a threshold. Treat the probability as the model output and the label as a product decision. This separation lets the same ranking support different costs and team capacities.

Check whether a score is a probability

Among many examples scored 0.8, roughly 80 percent should be positive before interpreting 0.8 literally. A model can rank cases well while being overconfident. Inspect a reliability curve and Brier score, then calibrate on independent data if expected cost or value uses the probability.

Use logistic regression as the baseline

Logistic regression maps a linear combination of features through a sigmoid. It is fast, regularizable, comparatively easy to explain, and can be well calibrated when assumptions are reasonable. It may not win on tabular data, but it sets a strong floor and exposes suspicious leakage when a simple model scores implausibly well.

Classification is not only binary

Binary chooses between two classes and usually centers one positive probability.
Multi-class chooses exactly one of K classes whose probabilities sum to one.
Multi-label assigns any subset of K labels, so each label needs a score and threshold.
Ordinal respects an order such as bad, okay, good and should penalize distant mistakes more strongly.

Fix the problem shape before the model

“Classify this request” can mean one routing department or several simultaneous topics, which require different targets and losses. Treating ordered grades as arbitrary regression numbers can impose a false equal distance. Contract how many answers one case may have, how unknown classes and abstentions work, and which errors cost more.

Compare at the real operating point

Compare models not only on ROC-AUC or PR-AUC but also on calibration, precision, recall, alert count, latency, and expected cost at the intended threshold. Log original score, applied threshold, policy version, model version, and final label separately so a later review can distinguish model and policy errors.

Code

Logistic regression with class weights and balanced output·python

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000, class_weight="balanced")),
])
pipe.fit(X_train, y_train)
probs = pipe.predict_proba(X_val)[:, 1]

Multi-label sigmoid head with OneVsRestClassifier·python

from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression

tags_model = OneVsRestClassifier(LogisticRegression(max_iter=1000)).fit(X_train, Y_train_multilabel)

Probabilities first, labels second

Check whether a score is a probability

Use logistic regression as the baseline

Classification is not only binary

Fix the problem shape before the model

Compare at the real operating point

Code

External links

Exercise

Progress

Comments 0