Random Forests

Average many unstable trees

A random forest grows trees on bootstrap samples of the training rows. Regression outputs are averaged; classification probabilities or votes are combined. One deep tree has high variance, but averaging trees that make different errors cancels many accidental splits.

Random features make trees different

If every split sees every feature, one dominant predictor can make all trees resemble one another. A forest considers a random feature subset at each split, lowering correlation among trees. Individual trees may become slightly weaker while the ensemble gains more from averaging.

Add trees until performance stabilizes

Increasing n_estimators usually reduces Monte Carlo variability but continually adds training time, prediction time, and memory. Plot validation or OOB performance from dozens to hundreds of trees and stop when improvement becomes operationally negligible.

Control the shape of each tree

max_features trades individual strength against diversity. max_depth and min_samples_leaf control noisy leaves; larger leaves may improve probability stability on small or noisy data. Defaults are baselines, not universal optima, so inspect fold variability and the train-validation gap.

OOB rows provide an inexpensive check

A bootstrap sample leaves out about 36.8% of rows on average. Combining predictions from trees that did not train on a row produces an out-of-bag estimate. It is useful for iteration, but does not replace group-aware or time-aware validation and an untouched final test set.

Validate probabilities and importance separately

A forest may rank cases well while its probabilities still need calibration. Impurity-based feature_importances_ can favor continuous or high-cardinality variables and distribute credit unpredictably among correlated features. Use held-out permutation importance with repeats, and do not interpret a ranking as causality.

A robust second baseline for tabular data

After logistic regression, a random forest is a strong candidate when nonlinear interactions matter and extensive tuning is undesirable. Compare it with boosting on identical folds and the business metric. Forests can be large and evaluate many trees per request, so include artifact size, throughput, latency, and calibration in the choice. Operational stability can beat a small leaderboard gain.

Code

Random forest with OOB and balanced classes·python

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=400, max_depth=None, max_features="sqrt",
    class_weight="balanced", oob_score=True, n_jobs=-1, random_state=7
)
rf.fit(X_train, y_train)
print("OOB score:", rf.oob_score_)

Permutation importance is more honest than feature_importances_·python

from sklearn.inspection import permutation_importance

perm = permutation_importance(rf, X_val, y_val, n_repeats=20, random_state=7, n_jobs=-1)
for name, score in sorted(zip(X_val.columns, perm.importances_mean), key=lambda x: -x[1])[:10]:
    print(f"{score:+.4f}  {name}")