Norms: How 'Big' Is a Vector?

Multiple Ways to Be Big

For a scalar, "size" is unambiguous — $∣ x ∣$ . For a vector you have choices, and each choice has a name and a use case:

L2 norm (Euclidean): $∥ v ∥_{2} = v_{1}^{2} + v_{2}^{2} + \dots + v_{n}^{2}$ — the straight-line length you'd measure with a ruler. Default norm when nobody specifies.
L1 norm (Manhattan): $∥ v ∥_{1} = ∣ v_{1} ∣ + ∣ v_{2} ∣ + \dots + ∣ v_{n} ∣$ — the taxi-cab distance, summing absolute values. Encourages sparsity in optimization (some components go all the way to zero).
L∞ norm (max): $∥ v ∥_{\infty} = max_{i} ∣ v_{i} ∣$ — just the biggest absolute value. Used when "the worst element" matters most.

Where Each Shows Up

L2 regularization (weight decay) — penalize $∥ w ∥_{2}^{2}$ . Common in deep learning. Encourages small, distributed weights.
L1 regularization (LASSO) — penalize $∥ w ∥_{1}$ . Drives many weights exactly to zero — automatic feature selection.
Gradient clipping — if $∥ g ∥_{2}$ exceeds a threshold, scale it down. Keeps training stable when gradients explode.

Normalizing a Vector

"Normalize" a vector usually means: divide by its L2 norm so the result has length 1. The direction is preserved; the magnitude becomes exactly 1. Then you can compare directions cleanly.

Norms are the rulers you choose. Different rulers measure different kinds of "big." Pick the one that matches what you're trying to constrain.

Code

Three rulers·python

import numpy as np

v = np.array([3.0, 4.0, 12.0])

print(np.linalg.norm(v, ord=2))     # 13.0  — Euclidean
print(np.linalg.norm(v, ord=1))     # 19.0  — Manhattan
print(np.linalg.norm(v, ord=np.inf))# 12.0  — Max

# Normalize to unit length
unit = v / np.linalg.norm(v)
print(np.linalg.norm(unit))         # 1.0 (or extremely close)

Exercise

Generate a random vector of length 1000 with np.random.randn(1000). Compute its L1 and L2 norms. Then divide the vector by its L2 norm and verify the result has L2 norm 1.

Hint

The L1 norm of a random Gaussian vector grows linearly with length; the L2 norm grows like sqrt(length). Different rulers, different stories about the same vector.

Norms: How 'Big' Is a Vector?

Multiple Ways to Be Big

Where Each Shows Up

Normalizing a Vector

Code

External links

Exercise

Progress

Comments 2

1000 길이 임의 벡터 생성

L1, L2 norm 계산

L2 norm으로 나눠서 단위 벡터 만들기

결과 벡터의 L2 norm 확인