covariance matrix

Topics

• covariance
• matrices

Square symmetric matrix capturing pairwise covariances between variables in a dataset. For random vector $X=[X_1,X_2,...,X_n{]}^T$, the covariance matrix $\Sigma$ has elements:

$${\Sigma }_{ij}=\text{Cov}(X_i,X_j)=\mathbb{E}[(X_i-{\mu }_i)(X_j-{\mu }_j)]$$

Diagonal contains variance values ${\Sigma }_{ii}=\text{Var}(X_i)$. Off-diagonals show how variables change together.

from numpy import array, cov
 
X = array([
[1, 5, 8],
[3, 5, 11],
[2, 4, 9],
[3, 6, 10],
[1, 5, 10]])
 
Sigma = cov(X.T)
 
# [1.0, 0.25, 0.75]
# [0.25, 0.5, 0.25]
# [0.75, 0.25, 1.2999999999999998]

Key Properties

• Symmetric: $\Sigma ={\Sigma }^T$
• Positive semi-definite: $v^T\Sigma v\ge 0$ for any vector $v$
• Measures linear relationships between variables
• Scale-dependent (affected by units of measurement)

The covariance matrix helps in understanding the relationships between variables and is used in methods like Principal Component Analysis (PCA) for dimensionality reduction.

Related

• variance of a vector
• correlation matrix (normalized covariance matrix)
• precision matrix (inverse of covariance matrix)