Log-Space Computing: The Underrated AI Skill

Combining Values in Log Space

Sometimes you need the log of a sum of probabilities while storing each probability as a log. Directly computing $lo g (e^{a} + e^{b})$ can underflow for very negative inputs and overflow for large positive inputs. The stable form is log-sum-exp:

$logsumexp (a, b) = m + lo g (e^{a - m} + e^{b - m}), m = max (a, b)$

Subtracting the maximum makes every exponent nonpositive, then adding it back preserves the exact algebra. Use scipy.special.logsumexp or torch.logsumexp.

Stable Softmax Uses the Same Shift

Softmax is $softmax (x_{i}) = e^{x_{i}} / \sum_{j} e^{x_{j}}$ . A naive implementation can overflow on large logits and underflow on very negative ones. Subtracting $max (x)$ changes neither the normalized probabilities nor their ordering and keeps exponentials in range.

When you need log-probabilities, use a library's log_softmax rather than computing softmax and then taking a log. PyTorch provides torch.nn.functional.log_softmax, and SciPy provides scipy.special.log_softmax.

Use a library logsumexp or log-softmax whenever the formula contains a log of summed exponentials. The naive algebra is correct over real numbers and fragile in floating point.

Code

logsumexp earns its name·python

import numpy as np
from scipy.special import logsumexp

# Numerically dangerous: very negative log-probs
log_probs = np.array([-1000., -1001., -1002.])

# Naive: log(sum(exp(...)))
naive = np.log(np.sum(np.exp(log_probs)))   # -inf — underflow

# Stable: logsumexp
stable = logsumexp(log_probs)
print(stable)                                # ~-999.59 — correct

Combining Values in Log Space

Stable Softmax Uses the Same Shift

Code

External links

Exercise

Progress

Comments 2

출력

manual : -999.5923940355556

scipy : -999.5923940355556

diff : 0.0