Visualizing Embeddings: Make the Geometry Real

Embeddings live in 768-12,288 dimensions. Humans live in 3. To bridge the gap, we project: t-SNE and UMAP are the two standard tools for collapsing high-dimensional points into 2D or 3D plots that preserve local neighborhood structure.

What you'll see in a well-trained embedding visualization: tight clusters of synonymous words, separate islands for different parts of speech, smooth gradients along consistent directions (gender, tense, formality). What you'll see in a poorly-trained or biased embedding: clusters that betray training-data biases, regions that conflate semantically distinct concepts, sparse outliers that didn't appear in training.

What this is for

Embedding visualization isn't just pretty pictures — it's a debugging tool. If your fine-tuned classifier confuses two categories, plot their embedding centroids. If they overlap or cluster too tightly, your input representation may be the bottleneck before you even reach the classification head. The same logic applies to RAG retrieval: if relevant and irrelevant documents end up in the same neighborhood, the retriever cannot tell them apart, and no amount of LLM cleverness downstream will fix it.

Code

Quick t-SNE plot·python

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

words = ["cat", "dog", "kitten", "puppy",
         "car", "truck", "vehicle", "engine",
         "happy", "joyful", "sad", "depressed"]
vectors = np.stack([get_embedding(w) for w in words])

xy = TSNE(n_components=2, perplexity=5).fit_transform(vectors)
plt.scatter(xy[:, 0], xy[:, 1])
for (x, y), w in zip(xy, words):
    plt.annotate(w, (x, y))
plt.savefig('embeddings.png')
# Animals cluster, vehicles cluster, emotions cluster — clearly.

Exercise

Build embeddings for 100 words spanning 5 categories you care about (animals, foods, emotions, programming concepts, anything). Project to 2D with both t-SNE and UMAP. Side-by-side plot. Note where the categorical clusters look clean and where they bleed into each other. Are the bleeds 'real' (meaningful overlap) or projection artifacts?

Visualizing Embeddings: Make the Geometry Real

What this is for

Code

External links

Exercise

Progress

Comments 0