Bayes' Rule: Updating Beliefs

The Equation

$P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}$

Bayes' rule relates the probability of a hypothesis after evidence to the likelihood of that evidence under the hypothesis, the prior probability, and a normalizing evidence term.

The Mental Model

Prior $P (A)$ : the probability assigned before observing $B$ .
Likelihood $P (B ∣ A)$ : how probable the observed evidence is if $A$ holds.
Posterior $P (A ∣ B)$ : the updated probability after observing $B$ .
Evidence $P (B)$ : the normalizer obtained across the possible causes of $B$ .

A Counterintuitive Test Result

Suppose 1% of a population has a disease, and a test has 99% sensitivity and 99% specificity. A positive result does not imply a 99% chance of disease. Among 10,000 people, about 99 of the 100 people with the disease test positive, while about 99 of the 9,900 healthy people produce false positives. That leaves about 198 positive results split nearly evenly between true and false positives, yielding a posterior near 50% under these assumptions.

Bayesian and Non-Bayesian ML Uses

Naive Bayes classifiers apply Bayes with conditional-independence assumptions.
Bayesian neural networks place distributions over weights or functions and update them with data.
Variational inference approximates difficult posterior distributions with tractable families.
Preference training methods such as RLHF and DPO update a policy using preference signals, but the resulting policy is not automatically a Bayesian posterior.

Bayes' rule is an exact relationship between prior, likelihood, evidence, and posterior. Calling any update a “Bayesian update” requires those probabilistic pieces, not just a before-and-after model.

Code

Counterintuitive disease testing·python

# Disease testing example
prior = 0.01            # 1% prevalence
sensitivity = 0.99      # P(positive | disease)
specificity = 0.99      # P(negative | no disease)

p_pos_given_disease   = sensitivity
p_pos_given_no_disease = 1 - specificity

# P(positive) — total probability
p_pos = p_pos_given_disease * prior + p_pos_given_no_disease * (1 - prior)

# Bayes
posterior = (p_pos_given_disease * prior) / p_pos
print(f"P(disease | positive test) = {posterior:.3f}")  # ~0.500

Exercise

Take the disease example. Change the prior to 50% (a much more common disease) and recompute the posterior. Why does the test accuracy now matter more for your conclusion?

Hint

With a 50% prior, a positive test is much stronger evidence — posterior ~99%. The lesson: priors dominate when the prior is extreme; evidence dominates when the prior is uncertain.

The Equation

The Mental Model

A Counterintuitive Test Result

Bayesian and Non-Bayesian ML Uses

Code

External links

Exercise

Progress

Comments 0