Dimensionality Reduction

~28 min · pca, umap, tsne

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Two reasons to reduce

Compression: many models speed up if you keep only the components that carry signal. Visualization: humans can read 2D scatter plots, not 200D. Pick the technique by which job you actually need.

The big three

PCA — linear, fast, deterministic. Excellent for compression and as a preprocessing step.
UMAP — non-linear, preserves both local and global structure reasonably well; today's go-to for visualization.
t-SNE — non-linear, beautiful local structure, can mislead about global distances; older but still useful for cluster inspection.

The cardinal mistake

Do not use UMAP or t-SNE coordinates as features for a downstream model. They are designed for visualization, not stable representations. PCA is fine; UMAP/t-SNE are not.

Code

PCA for compression with explained variance check·python

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

Xs = StandardScaler().fit_transform(X)
pca = PCA(n_components=0.95, svd_solver="full").fit(Xs)
print("components kept:", pca.n_components_)
print("explained variance:", np.cumsum(pca.explained_variance_ratio_)[-1].round(3))

UMAP for 2D visualization·python

import umap

reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=7)
embedding = reducer.fit_transform(Xs)
# embedding.shape == (n, 2) — for plotting only

External links

Exercise

Reduce your scaled feature matrix with PCA(n_components=0.95). Train your best model on the original features and on the PCA features. Compare CV score and train/predict latency. Decide whether the compression is worth shipping.