The Product Rule: Multiplication Becomes Addition

The Spell That Saved AI From Itself

The most important log identity for AI:

$lo g (A \cdot B) = lo g (A) + lo g (B)$

Read it again. Multiplication on the left, addition on the right. Logs let you trade hard operations (multiplying many tiny floats) for easy ones (adding their logs). This is not a notational convenience — it's an engineering miracle.

Numerical Underflow: The Bug This Identity Solves

Float32 represents numbers down to about $1 0^{- 38}$ . Multiply 100 probabilities of value 0.001 each:

$0.00 1^{100} = 1 0^{- 300} float32 0$

The result is mathematically tiny but not zero. Float32 rounds it to zero anyway. Now your loss is $lo g (0) = - \infty$ and your training crashes.

Take logs first: $lo g (0.00 1^{100}) = 100 \cdot lo g (0.001) \approx - 690$ . A perfectly normal float. Sum the log-probabilities instead of multiplying probabilities. Done.

Log-Likelihood Everywhere

This is why almost every probabilistic ML loss involves logs:

Cross-entropy loss: $- \sum_{i} y_{i} lo g (\overset{y}{^}_{i})$ — sums log-probabilities of true classes.
Maximum likelihood estimation: maximize $\sum_{i} lo g p (x_{i} ∣ θ)$ , not $\prod_{i} p (x_{i} ∣ θ)$ .
Language model training: minimize negative log-likelihood of the next token, summed over millions of tokens.

If you ever have to multiply many probabilities, take logs first and sum instead. The math is identical; the numerics are wildly better.

Code

Why logs aren't optional·python

import numpy as np

probs = np.array([0.001] * 100)

# The naive way — underflows
direct = np.prod(probs)
print(direct)            # 0.0  — float32/64 gave up

# The log-likelihood way — works perfectly
log_total = np.sum(np.log(probs))
print(log_total)         # -690.776...
print(np.exp(log_total)) # 1e-300 — only computable as a log

Why PyTorch loves logits·python

import torch

# PyTorch's cross-entropy loss already lives in log-space
logits = torch.tensor([[2.0, 1.0, 0.5]])    # raw model outputs
target = torch.tensor([0])                   # correct class index

loss = torch.nn.functional.cross_entropy(logits, target)
# Internally: log_softmax (logits) → pick the target index → negate
print(loss.item())                           # ~0.42

Exercise

Multiply 0.5 by itself 1000 times (use np.prod on an array of 0.5s). What does NumPy give you? Now compute 1000 * np.log(0.5). Then np.exp the result back. Why is the second route the only one that survives?

Hint

0.5^1000 ≈ 9.3e-302 — at the edge of float64. The log path computes -693.15, exp-ing it gives back the tiny number. The product route varies by NumPy version but is unreliable.

The Product Rule: Multiplication Becomes Addition

The Spell That Saved AI From Itself

Numerical Underflow: The Bug This Identity Solves

Log-Likelihood Everywhere

Code

External links

Exercise

Progress

Comments 2

방법 1

방법 2

비교