Anomaly Detection

~28 min · anomaly, outliers, isolation-forest

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Two flavors

Supervised anomaly detection has labels for known anomalies; treat it as classification with extreme imbalance. Unsupervised anomaly detection has no labels and assumes anomalies are rare and structurally different. The unsupervised case is what people usually mean.

Algorithms that matter

IsolationForest — fast, scales well, no distance assumptions.
LocalOutlierFactor — density-based, sensitive to local context.
One-class SVM — boundary around normal data; expensive on large datasets.
Autoencoder reconstruction error — for high-dimensional inputs (images, sensors).

Evaluation without labels

Without ground truth, evaluation is half art. Sample the top scores and have a human review. Track precision-at-K (of the top K alerts, how many were real). Wire the human reviews back as labels and gradually move toward supervised classification.

Code

IsolationForest as a fast first cut·python

from sklearn.ensemble import IsolationForest

iso = IsolationForest(
    n_estimators=300, contamination=0.01, random_state=7, n_jobs=-1
).fit(X_train)
scores = -iso.score_samples(X_new)  # higher = more anomalous

Top-K alerts for the on-call team·python

import numpy as np

K = 100
top_idx = np.argsort(-scores)[:K]
alerts = X_new.iloc[top_idx].assign(score=scores[top_idx])
alerts.to_csv("alerts/today.csv", index=False)

External links

Exercise

Run IsolationForest on your dataset with contamination=0.01. Sample the top 50 anomalies and label them as real, false, or ambiguous. Compute precision@50. Decide whether the model is ready for an on-call rotation.