Bayes' Rule: Updating Beliefs

The Equation

$P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}$

Read aloud: "the probability of A given B equals the probability of B given A, times the probability of A, divided by the probability of B." That's Bayes' rule. It tells you how to update a belief about A when you observe B.

The Mental Model

Prior $P (A)$ — what you believed about A before seeing the evidence.
Likelihood $P (B ∣ A)$ — how likely the evidence B is, assuming A were true.
Posterior $P (A ∣ B)$ — your updated belief about A after seeing B.
Evidence $P (B)$ — a normalizer that ensures probabilities sum to 1.

The Classic Counter-Intuition Example

1% of the population has a disease. A test for the disease is 99% accurate (99% true positive rate, 99% true negative rate). You test positive. What's the probability you actually have it?

Most people guess 99%. Bayes says ~50%. The reason: the prior is 1% — diseases are rare — and the false-positive rate (1%) applied to the 99% of healthy people produces almost as many false positives as the 99% true positive rate produces true positives. Updating from a low prior takes more evidence than your gut estimates.

Bayes in ML

Naive Bayes classifiers — apply Bayes directly with independence assumptions.
Bayesian neural networks — treat weights as distributions, update with data.
Variational inference — approximate intractable posteriors with simpler distributions.
RLHF / DPO — modern LLM training updates a prior policy toward a posterior aligned with human preferences.

Bayes' rule is the math of belief update. Whenever you reason "I thought X, but I just learned Y, so now I think Z," you're approximating Bayes — usually badly.

Code

Counterintuitive disease testing·python

# Disease testing example
prior = 0.01            # 1% prevalence
sensitivity = 0.99      # P(positive | disease)
specificity = 0.99      # P(negative | no disease)

p_pos_given_disease   = sensitivity
p_pos_given_no_disease = 1 - specificity

# P(positive) — total probability
p_pos = p_pos_given_disease * prior + p_pos_given_no_disease * (1 - prior)

# Bayes
posterior = (p_pos_given_disease * prior) / p_pos
print(f"P(disease | positive test) = {posterior:.3f}")  # ~0.500

Exercise

Take the disease example. Change the prior to 50% (a much more common disease) and recompute the posterior. Why does the test accuracy now matter more for your conclusion?

Hint

With a 50% prior, a positive test is much stronger evidence — posterior ~99%. The lesson: priors dominate when the prior is extreme; evidence dominates when the prior is uncertain.

Bayes' Rule: Updating Beliefs

The Equation

The Mental Model

The Classic Counter-Intuition Example

Bayes in ML

Code

External links

Exercise

Progress

Comments 0