Decision Trees

~28 min · trees, interpretable

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

How a tree learns

A decision tree recursively splits the data on a feature value to minimize an impurity criterion (Gini for classification, MSE for regression). Each leaf gets the majority label or mean target. Trees natively handle missingness, mixed types, monotone and non-monotone relationships, and interactions, which is why they punch above their weight on tabular data.

Strengths and weaknesses

Strength: easy to inspect; one tree can be drawn on a whiteboard for product reviews.
Strength: no scaling, no encoding pain, no normality assumptions.
Weakness: single trees overfit aggressively at high depth.
Weakness: small data perturbations produce wildly different trees.

How to use them

Use a single shallow tree (depth 3-5) as an inspectable baseline and as a stakeholder explanation tool. For prediction quality, almost always graduate to random forests or gradient boosting; the bias-variance tradeoff is solved by ensembling.

Code

Train and visualize a shallow tree·python

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(max_depth=4, class_weight="balanced", random_state=7)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=list(X_train.columns)))

Plot the tree in a notebook·python

from sklearn.tree import plot_tree
import matplotlib.pyplot as plt

plt.figure(figsize=(14, 7))
plot_tree(tree, filled=True, feature_names=list(X_train.columns), max_depth=3)
plt.show()

External links

Exercise

Train a max_depth=4 decision tree on your dataset. Print it with export_text. Show it to a non-technical stakeholder and ask if the splits match their domain intuition. Note any surprises as data hypotheses to investigate.