Random Forests

The bagging recipe

A random forest grows many decision trees on bootstrap samples, each tree splitting on a random subset of features. Predictions are averaged (regression) or voted (classification). Bagging reduces variance dramatically while keeping bias roughly the same as a single tree. Random feature subsetting decorrelates the trees so the variance reduction is real.

What to tune (and what not to)

n_estimators — more is usually better until you run out of compute. Diminishing returns past a few hundred.
max_features — sqrt for classification, 1/3 for regression are sane defaults.
max_depth, min_samples_leaf — control individual tree complexity; defaults work surprisingly often.
Out-of-bag (OOB) score — free estimate of generalization; turn it on with oob_score=True.

When to reach for it

Random forests are the "just works" tabular baseline above logistic regression. Use them when you want strong out-of-the-box performance with minimal tuning and you can tolerate slightly larger memory footprints. For the very top of the leaderboard, gradient boosting usually edges them out.

Code

Random forest with OOB and balanced classes·python

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=400, max_depth=None, max_features="sqrt",
    class_weight="balanced", oob_score=True, n_jobs=-1, random_state=7
)
rf.fit(X_train, y_train)
print("OOB score:", rf.oob_score_)

Permutation importance is more honest than feature_importances_·python

from sklearn.inspection import permutation_importance

perm = permutation_importance(rf, X_val, y_val, n_repeats=20, random_state=7, n_jobs=-1)
for name, score in sorted(zip(X_val.columns, perm.importances_mean), key=lambda x: -x[1])[:10]:
    print(f"{score:+.4f}  {name}")

The bagging recipe

What to tune (and what not to)

When to reach for it

Code

External links

Exercise

Progress

Comments 0