Mental Pictures: Vector, Matrix, Tensor

Geometry first, indices second

The fastest way to forget what a tensor 'is' is to read its definition as 'an N-dimensional array.' That's the storage view. The intuition that helps you debug a model is geometric:

Vector (1-D) → an arrow with length (norm) and direction (feature mix). A 1024-D embedding is a single arrow in 1024-dimensional space. 'Similar concepts' = arrows pointing the same way.
Matrix (2-D) → a stack of those arrows. Rows or columns, depending on layout. The choice between row-major and column-major is exactly which axis is contiguous in memory.
Tensor (3-D+) → 'a matrix per time-step, per batch, per attention head.' Compute-wise it's still batches of matrices going through GEMM; the extra dims are bookkeeping.

This matters because every shape mismatch you ever debug is a question about which axis is which. 'Why is my attention output (32, 16, 64, 512) not (32, 512, 16, 64)?' is solved by holding the geometric picture: batch × heads × seq × head_dim, and remembering that the GEMM happened along the last two axes.

Code

Build the four shapes you'll see daily·python

import numpy as np

# Vector — a 1024-D arrow (single embedding)
embedding = np.random.randn(1024)
magnitude = np.linalg.norm(embedding)            # length
direction = embedding / magnitude                # unit arrow

# Matrix — a batch of 32 embeddings stacked as rows
embeddings_batch = np.random.randn(32, 1024)

# Tensor — 16 sequences, 512 tokens each, 1024 dims
sequence_data = np.random.randn(16, 512, 1024)

# 4-D tensor — multi-head attention input
# (batch, heads, seq, head_dim)
attention_input = np.random.randn(32, 16, 512, 64)

Cosine similarity — the dot-product as geometric meaning·python

def cosine(u, v):
    return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))

a = np.array([1.0, 1.0, 0.0])    # arrow pointing 'up-right'
b = np.array([1.0, 0.9, 0.1])    # almost the same direction
c = np.array([-1.0, -1.0, 0.0])  # opposite direction
d = np.array([1.0, -1.0, 0.0])   # perpendicular to a

print(cosine(a, b))   # ~0.99 — nearly identical concept
print(cosine(a, c))   # -1.0  — opposite
print(cosine(a, d))   #  0.0  — orthogonal, unrelated

Exercise

Take your favorite embedding model (sentence-transformers, OpenAI ada, anything with a Python wrapper) and embed three sentences: (1) 'Cats love sunshine.' (2) 'Sunbathing kittens are happy.' (3) 'Stock market crashed.' Compute pairwise cosine similarities. You should see (1,2) ≈ 0.7+, both ≪ to (3). The numbers don't need to match anyone else's — but if (1,2) isn't clearly closer than either is to (3), check that you're using cosine and not Euclidean.

Mental Pictures: Vector, Matrix, Tensor

Geometry first, indices second

Code

External links

Exercise

Progress

Comments 0