Joint: Two Coins Are Not One Coin Twice

"The arithmetic of probability is multiplication, not addition. Addition is for things that can't both happen. Multiplication is for things that can."

Three Words That Look Similar and Mean Three Different Things

Three words show up in every probability sentence and most readers blur them:

Marginal — P(A) on its own. The probability of A ignoring everything else. "What fraction of people are over 60?"
Joint — P(A and B). The probability of both A and B happening together. "What fraction of people are over 60 AND have had a heart attack?"
Conditional — P(A | B). Probability of A inside the universe where B is true. "Among people over 60, what fraction have had a heart attack?"

Three different questions. Three different numbers. They are linked by one identity: P(A and B) = P(A | B) × P(B). The joint is the conditional times the marginal of the condition.

The 'OR' Versus 'AND' Trap

The citizen mistake is reaching for addition when the event is a conjunction. "It will probably rain (60%) AND I'll probably forget my umbrella (30%), so the chance of getting wet is 60% + 30% = 90%." Wrong on two counts.

For an AND (joint), you multiply. If those two were independent: 0.60 × 0.30 = 0.18. There's an 18% chance of both, not 90%.

For an OR (union), you add — and then you subtract the overlap so you don't count it twice. P(A or B) = P(A) + P(B) − P(A and B).

The intuition trap is the word 'and' sounding like 'sum' in English. It is not. In probability, AND is product (with a conditional twist when events aren't independent); OR is sum-minus-overlap.

The Chain Rule — Joint When Things Aren't Independent

When A and B are independent, P(A and B) = P(A) × P(B). Clean.

When they're not — which, as the previous lesson reminded us, is most of the time — you need the chain rule: P(A and B) = P(A | B) × P(B). Read it as: "first you survived the B-event, with probability P(B); then, given you survived, A happens with probability P(A | B)". The chain rule is conditional probability being honest about the coupling.

Two Coins Are Not One Coin Twice

The joint of two events lives in a different space than either marginal. Two fair coins flipped together don't give you 'a coin flipped twice with double the heads probability.' They give you a 2-by-2 joint distribution: HH, HT, TH, TT — each with probability 0.25. The arithmetic of probability happens in the joint space, not in the marginals stacked side by side.

Where This Matters Soon

The likelihood function in Bayes' rule is exactly a joint probability of many observations, computed by multiplying conditionals. Maximum likelihood estimation, the workhorse of modern statistics and machine learning, is the art of choosing a hypothesis that maximizes one big joint probability. If you can multiply correctly here, you can read a likelihood. If you can't, you'll get every later technique wrong on the inside while the outside looks fine.

Code

Joint, marginal, and the chain rule, side by side·python

import numpy as np
rng = np.random.default_rng(2)

# A 2-by-2 joint distribution: rain (R) vs umbrella forgotten (F).
# Suppose these two are NOT independent — you forget the umbrella more often
# when the morning is sunny (because you don't bother grabbing it).
N = 100_000
rain     = rng.random(N) < 0.30   # P(R)  = 0.30
# Forgetting is correlated with NOT raining (you only grab the umbrella when
# the sky looks threatening).
forgot   = np.where(rain, rng.random(N) < 0.10, rng.random(N) < 0.50)

P_R  = rain.mean()
P_F  = forgot.mean()
P_RF = (rain & forgot).mean()
P_F_given_R = forgot[rain].mean()

print(f"Marginal P(rain)              = {P_R:.3f}")
print(f"Marginal P(forgot umbrella)   = {P_F:.3f}")
print(f"Joint    P(rain AND forgot)   = {P_RF:.3f}")
print(f"Chain    P(forgot|rain)*P(rain) = {P_F_given_R * P_R:.3f}  <- matches joint")
print(f"Product  P(rain)*P(forgot)    = {P_R * P_F:.3f}            <- WRONG: assumes independence")

# The chain-rule version matches the empirical joint.
# The product-of-marginals version is off — that's the cost of assuming
# independence when there's actually coupling.

Exercise

Estimate three numbers for the morning weather example with your own city: marginal P(rain on a random morning), conditional P(you carry an umbrella | it's raining), and joint P(rain AND you carry an umbrella). Check that your joint is at most equal to your marginal of rain — if not, redo the conditional. The joint can never exceed either marginal.

Hint

Joint is always 'at most as common as either ingredient on its own.' If joint > marginal, you've mixed up the formula direction.

Three Words That Look Similar and Mean Three Different Things

The 'OR' Versus 'AND' Trap

The Chain Rule — Joint When Things Aren't Independent

Two Coins Are Not One Coin Twice

Where This Matters Soon

Code

External links

Exercise

Progress

Comments 0