Probability vs Likelihood: The Distinction Worth Knowing

Two Words That Aren't Synonyms

In casual English, "probability" and "likelihood" mean the same thing. In statistics they don't, and the distinction matters.

	Question	Variable
Probability $P (D ∣ θ)$	Given parameters $θ$ , how likely is the data $D$ ?	data varies, parameters fixed
Likelihood $L (θ ∣ D)$	Given the data $D$ , how plausible are the parameters $θ$ ?	data fixed, parameters vary

Same formula, different perspective. Probability looks forward (model → data); likelihood looks backward (data → model).

Why ML Cares

Maximum Likelihood Estimation (MLE) is the foundation of model training. The principle: the best parameters are the ones that make the observed data most plausible. Mathematically: $\hat{θ} = ar g max_{θ} L (θ ∣ D)$ . In practice you minimize negative log-likelihood, which is what cross-entropy loss really is.

Every time you train a classifier, you're solving an MLE problem in disguise.

Softmax: Logits → Probability Distribution

Neural network classifiers output raw scores called logits. To turn them into probabilities (non-negative, summing to 1), apply softmax:

$softmax (x_{i}) = \frac{e ^{x_{i}}}{\sum _{j} e ^{x_{j}}}$

Subtracting $max (x)$ for numerical stability before exponentiating is now standard.

Probability fixes the model and varies the data. Likelihood fixes the data and varies the model. Training is searching parameter-space for the model that makes the data most plausible.

Code

Softmax — logits to probabilities·python

import numpy as np

def softmax(logits):
    shifted = logits - logits.max()        # stability
    exp = np.exp(shifted)
    return exp / exp.sum()

logits = np.array([2.0, 1.0, 0.5, -1.0])
probs = softmax(logits)
print(probs)                                # [0.620, 0.228, 0.139, 0.013]
print(probs.sum())                          # 1.0 — proper distribution

Exercise

Take logits [3.0, 1.0, 0.0]. Apply softmax. Then add 1000 to every logit and apply softmax again. Why is the result the same? What does this tell you about absolute logit values?

Hint

Softmax is invariant to constant shifts: softmax(x + c) = softmax(x) because the constant cancels. Only the *differences* between logits matter, not their absolute values.

Probability vs Likelihood: The Distinction Worth Knowing

Two Words That Aren't Synonyms

Why ML Cares

Softmax: Logits → Probability Distribution

Code

External links

Exercise

Progress

Comments 0