Log-Space Computing: The Underrated AI Skill

Living in Log Space

Once you start summing log-probabilities, you sometimes need to combine log-probabilities — for example, the log of a sum of probabilities. The naive way ( $lo g (e^{a} + e^{b})$ ) hits underflow when $a, b$ are very negative. The fix: log-sum-exp.

$logsumexp (a, b) = m + lo g (e^{a - m} + e^{b - m}) where m = max (a, b)$

Subtracting the max keeps the exponentials in a sane range. Adding it back at the end recovers the right answer. scipy.special.logsumexp and torch.logsumexp handle it for you.

Why Softmax Is Really Log-Softmax in Disguise

Softmax: $softmax (x_{i}) = \frac{e ^{x_{i}}}{\sum _{j} e ^{x_{j}}}$ . Naive implementation underflows when $x_{j}$ is very negative. Modern frameworks subtract $max (x)$ first — same trick as log-sum-exp.

If you ever need log(softmax(x)), never compute it as np.log(softmax(x)). Use log_softmax(x) directly — it's numerically stable. PyTorch and NumPy/SciPy provide it.

Whenever your math involves "log of a sum of exponentials," use the library's logsumexp. The naive version is correct in theory and wrong in float32.

Code

logsumexp earns its name·python

import numpy as np
from scipy.special import logsumexp

# Numerically dangerous: very negative log-probs
log_probs = np.array([-1000., -1001., -1002.])

# Naive: log(sum(exp(...)))
naive = np.log(np.sum(np.exp(log_probs)))   # -inf — underflow

# Stable: logsumexp
stable = logsumexp(log_probs)
print(stable)                                # ~-999.59 — correct

Log-Space Computing: The Underrated AI Skill

Living in Log Space

Why Softmax Is Really Log-Softmax in Disguise

Code

External links

Exercise

Progress

Comments 2

출력

manual : -999.5923940355556

scipy : -999.5923940355556

diff : 0.0