vector norm

Topics

• distance metrics
• linear algebra

Vector norm: function that assigns non-negative length/size to a vector. For vector $\mathbf{v}$ in ${\mathbb{R}}^n$, $p$-norm written as $\Vert \mathbf{v}{\Vert }_p$, satisfying:

1. Non-negativity: $\Vert \mathbf{v}{\Vert }_p\ge 0$, $=0$ only if $\mathbf{v}=0$
2. Absolute homogeneity: $\Vert \alpha \mathbf{v}{\Vert }_p=|\alpha |\cdot \Vert \mathbf{v}{\Vert }_p$ for any scalar $\alpha$
3. Triangle inequality: $\Vert \mathbf{u}+\mathbf{v}{\Vert }_p\le \Vert \mathbf{u}{\Vert }_p+\Vert \mathbf{v}{\Vert }_p$
4. Definiteness: $\Vert \mathbf{v}{\Vert }_p=0⟺\mathbf{v}=0$

L1 Norm (Manhattan Norm)

$$\Vert \mathbf{v}{\Vert }_1=\sum _{i=1}^n|v_i|$$

• Notation: $\Vert \mathbf{v}{\Vert }_1$
• Definition: Sum of absolute components
• Interpretation: “Taxicab”/“Manhattan” distance from origin
• Use in ML: As a regularizer, it encourages sparsity (L1 regularization)

L₂ Norm (Euclidean Norm)

$$\Vert \mathbf{v}{\Vert }_2=\sqrt{\sum _{i=1}^n{v}_i^2}$$

• Notation: $\Vert \mathbf{v}{\Vert }_2$ (often just $\Vert \mathbf{v}\Vert$)
• Definition: Square root of sum of squares
• Interpretation: Standard Euclidean distance from origin
• Use in ML: Common regularizer (L2 regularization), ridge regression, weight decay

Max (∞) Norm

$$\Vert \mathbf{v}{\Vert }_{\infty }=\underset{1\le i\le n}{\max }|v_i|$$

• Notation: $\Vert \mathbf{v}{\Vert }_{\infty }$
• Definition: Maximum absolute component
• Interpretation: Greatest distance along any single coordinate axis
• Use in ML: Bounding weights (max-norm regularization in neural nets)

General p-Norms and Properties

$$\Vert \mathbf{v}{\Vert }_p=(\sum _{i=1}^n|v_i{|}^p{)}^{1/p}$$

• $p$-norm for $1\le p<\infty$:
• As $p\to \infty$, $\Vert \mathbf{v}{\Vert }_p\to \Vert \mathbf{v}{\Vert }_{\infty }$
• All satisfy triangle inequality and definiteness

from math import inf
from numpy import array
from numpy.linalg import norm
 
 
a = array([1, 2, 3])
 
l1 = norm(a, 1)
l2 = norm(a, 2)
l_inf = norm(a, inf)

Why Vector Norms Matter in Machine Learning

• Regularization: Adding $\Vert \mathbf{w}{\Vert }_1$ or $\Vert \mathbf{w}{\Vert }_2$ penalties controls model complexity
• Distance Measures: Nearest-neighbor classifiers, clustering rely on L1/L2 distances
• Optimization Geometry: Norms define contours, influence convergence of gradient-based methods

Related