PCA

Topic

• dimensionality reduction
• linear algebra

Principal component analysis is a fundamental technique for reducing dimensionality in data while preserving its structure. It transforms correlated variables into linearly uncorrelated principal components through orthogonal transformation. Useful for: data visualization, feature extraction etc.

Core idea

Given data matrix $X$ with $n$ samples and $m$ features, PCA finds new axes (principal components) that maximize variance. These axes are eigenvectors of covariance matrix $C=\frac{1}{n}{X}^{T}X$ where $X$ is mean-centered. The covariance matrix can be computed easily if you understand the calculation behind variance of a matrix.

Key steps

1. Center data by subtracting mean from each feature
2. Compute covariance matrix of centered data
3. Perform eigendecomposition of covariance matrix
4. Sort eigenvectors by eigenvalues in descending order
5. Select top $k$ eigenvectors as principal components
6. Project data onto new subspace using $Y=XW$ where $W$ contains selected eigenvectors

Implementation options

# Using numpy
cov_matrix = np.cov(X_centered.T)
eigenvals, eigenvecs = np.linalg.eig(cov_matrix)
 
# project data
P = eigenvecs.T.dot(X_centered.T)
 
print(P.T)

# Using scikit-learn
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_transformed = pca.fit_transform(X)
 
# access values and vectors
print(pca.components_)
print(pca.explained_variance_)

Warning

PCA is sensitive to outliers and assumes linear relationships. For non-linear dimensionality reduction, consider manifold learning techniques.

Related

• SVD