Rules, Heuristics, Statistics, and ML

Four overlapping tools

Rules encode known logic that almost always holds. Heuristics encode useful shortcuts that work most of the time. Statistics estimates relationships and uncertainty under modeling assumptions. Machine learning fits parameters that minimize a measured prediction loss. They overlap, but they are not interchangeable, and reaching for ML when a rule would do is a common professional mistake.

Tool	Best fit	Risk
Rule	Stable known logic, audit required	Brittle on rare exceptions
Heuristic	Fast approximation, low stakes	Silent bias accumulates
Statistics	Inference and uncertainty	Wrong causal story
ML	Prediction from labeled examples	Leakage and drift

Choosing the right one

Pick the simplest tool that absorbs the variation in the problem. If the rule is stable and inspectable, do not replace it with a model just to feel modern. If business stakeholders cannot define the target without a long meeting, no model will rescue them.

Combining them

Real systems mix all four. A fraud system might use rules for known patterns, a heuristic threshold for high-velocity accounts, a statistical baseline for expected behavior, and an ML model for the unknown unknowns. The interesting design question is which layer owns which case.

Code

Decision boundary: when does ML earn its keep?·python

def choose_tool(problem):
    if problem.rule_is_stable and problem.exceptions_are_rare:
        return "rule"
    if problem.needs_uncertainty and problem.sample_size_is_small:
        return "statistics"
    if problem.labels_exist and problem.mistakes_are_measurable:
        return "ml"
    return "clarify_problem_before_modeling"

Rule + ML hybrid: rule wins, ML covers the long tail·python

def predict(row, rule_fn, model):
    rule_decision = rule_fn(row)
    if rule_decision is not None:
        return rule_decision  # auditable path
    return model.predict_proba([row.features])[0, 1] >= 0.42

Four overlapping tools

Choosing the right one

Combining them

Code

External links

Exercise

Progress

Comments 0