Causality Limits

~24 min · causality, limits

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Prediction is not intervention

An ML model trained on observational data answers "given the world as it is, what is likely?" It does not answer "what would happen if I changed feature X?" Acting on a feature that is merely a predictor — not a cause — can move the prediction without moving the outcome.

Common confusions

"Tickets last 30 days" predicts churn → reducing tickets does not necessarily reduce churn (both are caused by frustration).
"Login count" predicts retention → forcing logins does not retain users.
"Discount usage" predicts loyalty → giving everyone discounts may train customers to expect them.

When you actually need causality

If the question is "what should we do?", you need experiments (A/B tests), causal inference (DiD, instrumental variables), or uplift modeling. ML predictions can prioritize who to test on, but the test answers the causal question, not the model.

Code

Uplift framing: predict the lift, not the outcome·python

# Two-model uplift: train one model on treated, one on untreated
from lightgbm import LGBMClassifier

treated = LGBMClassifier().fit(X[treatment == 1], y[treatment == 1])
control = LGBMClassifier().fit(X[treatment == 0], y[treatment == 0])
uplift = treated.predict_proba(X)[:, 1] - control.predict_proba(X)[:, 1]
# Target users with the highest uplift, not the highest baseline probability

DoWhy for causal sanity checks·python

import dowhy
from dowhy import CausalModel

model = CausalModel(
    data=df, treatment="discount_used",
    outcome="renewed", common_causes=["plan_tier", "tenure_days"]
)
estimand = model.identify_effect()
estimate = model.estimate_effect(estimand, method_name="backdoor.linear_regression")
print(estimate)

External links

Exercise

Pick one feature your team plans to act on ("increase logins", "reduce ticket count"). Sketch the causal graph: what causes the feature, what causes the target, what causes both. Decide whether an A/B test is needed before acting.