Dimensionality Reduction

Dimensionality reduction has two jobs

It can compress a feature matrix for modeling or project it into two dimensions for human inspection. Those goals require different evidence. An attractive picture may discard predictive information, while components that preserve variance may remain hard to interpret.

PCA finds linear directions of maximum variance

Principal component analysis produces orthogonal components and a consistent transform for new rows. Scale inputs when units differ, because large-unit features can dominate. Fit PCA on each training fold and transform validation or test data afterward.

Explained variance guides compression, not truth

Plot cumulative explained variance and identify a practical range of component counts. Then select the dimension using downstream validation and operating constraints. Choosing it after observing the test set leaks information, and high retained variance does not guarantee preservation of the signal the task needs.

UMAP projects neighborhood structure

UMAP creates a nonlinear neighborhood representation useful for exploration. Distances, island spacing, and empty regions depend on the input metric, neighborhood size, minimum distance, and initialization. Run plausible settings rather than treating one projection as a map of reality.

t-SNE emphasizes local neighbors

t-SNE can reveal local groupings, but global distances and apparent cluster sizes should not be read literally. Different perplexities and seeds can change the arrangement while preserving some nearby relationships.

Do not confuse a visual coordinate with a production feature

UMAP can transform new data in supported workflows, but that does not make every projection stable enough for a downstream model. If coordinates become features, version the fitted reducer, validate out of sample, test seed and parameter sensitivity, and monitor drift. PCA has the same train-only fitting boundary even though its mapping is linear.

Connect uncertainty to the original cases

Color projections only with metadata not used to construct them and inspect source examples behind every apparent island. Repeat across seeds and plausible parameters. A two-dimensional picture is an instrument for generating hypotheses, not proof that the original data contains discrete classes.

Code

PCA for compression with explained variance check·python

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

Xs = StandardScaler().fit_transform(X)
pca = PCA(n_components=0.95, svd_solver="full").fit(Xs)
print("components kept:", pca.n_components_)
print("explained variance:", np.cumsum(pca.explained_variance_ratio_)[-1].round(3))

UMAP for 2D visualization·python

import umap

reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=7)
embedding = reducer.fit_transform(Xs)
# embedding.shape == (n, 2) — for plotting only