Norms: How 'Big' Is a Vector?

Several Rulers for Vector Size

A scalar's size is its absolute value, but vectors admit several useful norms:

L2 norm: $∥ v ∥_{2} = \sum_{i} v_{i}^{2}$ , the Euclidean distance from the origin.
L1 norm: $∥ v ∥_{1} = \sum_{i} ∣ v_{i} ∣$ , connected to distance along an axis-aligned grid.
L∞ norm: $∥ v ∥_{\infty} = max_{i} ∣ v_{i} ∣$ , determined by the largest component.

Unqualified “norm” often means L2, but APIs differ. Changing the norm changes the geometry of what counts as large or nearby.

Normalization and Regularization Are Different Operations

Vector normalization divides by a norm, as in $v /∥ v ∥_{2}$ , to produce unit length. It is undefined for the zero vector without an explicit convention.
L1 regularization adds $λ ∥ w ∥_{1}$ to an objective and tends to favor sparse solutions; LASSO is the standard example.
L2 regularization adds a penalty such as $λ ∥ w ∥_{2}^{2}$ to discourage large weights. Traditional SGD weight decay is closely related, while optimizers such as AdamW decouple weight decay from the gradient-based penalty.
Gradient clipping rescales a gradient when its norm exceeds a threshold to reduce exploding updates.

The same norm can be used in distinct operations. State whether you are measuring, dividing by the measurement, adding a penalty, or clipping an update.

A norm is a ruler. The operation is not fully specified until you say what you do with that measurement.

Code

Three rulers·python

import numpy as np

v = np.array([3.0, 4.0, 12.0])

print(np.linalg.norm(v, ord=2))     # 13.0  — Euclidean
print(np.linalg.norm(v, ord=1))     # 19.0  — Manhattan
print(np.linalg.norm(v, ord=np.inf))# 12.0  — Max

# Normalize to unit length
unit = v / np.linalg.norm(v)
print(np.linalg.norm(unit))         # 1.0 (or extremely close)

Exercise

Generate a random vector of length 1000 with np.random.randn(1000). Compute its L1 and L2 norms. Then divide the vector by its L2 norm and verify the result has L2 norm 1.

Hint

The L1 norm of a random Gaussian vector grows linearly with length; the L2 norm grows like sqrt(length). Different rulers, different stories about the same vector.

Several Rulers for Vector Size

Normalization and Regularization Are Different Operations

Code

External links

Exercise

Progress

Comments 2

1000 길이 임의 벡터 생성

L1, L2 norm 계산

L2 norm으로 나눠서 단위 벡터 만들기

결과 벡터의 L2 norm 확인