The Product Rule: Multiplication Becomes Addition

Move Products into Sums

For positive $A$ and $B$ ,

$lo g (A \cdot B) = lo g (A) + lo g (B)$

This is the exponential rule $b^{u + v} = b^{u} b^{v}$ read backward. When a probability calculation contains a long product, the identity lets us work with a sum of log-probabilities instead.

Underflow Depends on the Data Type

For example, $0.00 1^{100} = 1 0^{- 300}$ . That is far below float32's range and underflows to zero, while float64 can still represent a nonzero result. The computer did not universally “give up”; a particular representation ran out of range.

Log space stores $100 lo g (0.001) \approx - 690.8$ , which is easy to represent. Exponentiating later can still underflow if the requested probability is outside the destination type's range, so comparisons, optimization, and accumulation should remain in log space as long as practical.

Where Log-Likelihood Appears

Maximum likelihood estimation maximizes a sum of observation log-probabilities instead of their product.
Cross-entropy for one-hot classification reduces to the negative log-probability of the target class.
Language models train on next-token negative log-likelihood and derive perplexity from mean log loss.

Turn long products of probabilities into sums of log-probabilities. The mathematics is equivalent, but the useful numerical range is much wider.

Code

Why logs aren't optional·python

import numpy as np

probs32 = np.full(100, 0.001, dtype=np.float32)
probs64 = np.full(100, 0.001, dtype=np.float64)

print(np.prod(probs32))  # 0.0: underflow in float32
print(np.prod(probs64))  # about 1e-300: still representable in float64

log_total = np.sum(np.log(probs32), dtype=np.float64)
print(log_total)         # about -690.776
print(np.exp(log_total)) # about 1e-300: recoverable in float64 here

# A longer product eventually underflows in float64 too; its log still survives.
deeper_log_total = 400 * np.log(0.001)
print(deeper_log_total)          # about -2763.102
print(np.exp(deeper_log_total))  # 0.0: underflow even in float64
# Keep the log value as long as the computation permits.

Why PyTorch loves logits·python

import torch

# PyTorch's cross-entropy loss already lives in log-space
logits = torch.tensor([[2.0, 1.0, 0.5]])    # raw model outputs
target = torch.tensor([0])                   # correct class index

loss = torch.nn.functional.cross_entropy(logits, target)
# Internally: log_softmax (logits) → pick the target index → negate
print(loss.item())                           # ~0.42

Exercise

Create arrays of 1,500 values equal to 0.5 with dtype=np.float32 and dtype=np.float64. Compare their direct products with 1500 * np.log(0.5).

Hint

0.5^1500 is below float64's normal range. Direct multiplication and final exponentiation may become zero, while the log value of about -1039.7 remains representable and comparable.

Move Products into Sums

Underflow Depends on the Data Type

Where Log-Likelihood Appears

Code

External links

Exercise

Progress

Comments 2

방법 1

방법 2

비교