Embeddings: Text Becomes Numbers

The embedding contract

An embedding model is a neural network that takes a string and returns a fixed-length list of floats called a vector. The model is trained so that texts with similar meaning produce vectors that are close together. You don't choose what each dimension means — the model learned that during pretraining on billions of documents.

Two things matter from day one:

The same model encodes both queries and documents. Mixing models breaks similarity completely — the vectors live in different spaces.
Vectors have a dimension. Common sizes are 384 (small, fast), 768 (workhorse), 1024–1536 (production sweet spot), and 3072 (top-tier OpenAI / Voyage). More dimensions can capture more nuance but cost more storage and slower comparisons.

Run an embedding locally

The easiest first step is sentence-transformers with a small model. It runs on CPU, downloads weights once, and returns NumPy arrays you can inspect. Once you can print a vector and see that two related sentences land closer than two unrelated ones, the rest of the course is just engineering.

Code

First embeddings with sentence-transformers·python

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('BAAI/bge-small-en-v1.5')   # 384-dim, ~30 MB
vecs = model.encode([
    'How do I cancel my annual plan?',
    'I want to end my yearly subscription.',
    'What time does the moon rise tonight?',
])
print(vecs.shape)         # (3, 384)
print(np.linalg.norm(vecs, axis=1))   # check normalization

Confirm related sentences are nearer than unrelated ones·python

from numpy import dot
from numpy.linalg import norm

def cos(a, b):
    return dot(a, b) / (norm(a) * norm(b))

print('related   :', cos(vecs[0], vecs[1]))   # ~0.85+
print('unrelated :', cos(vecs[0], vecs[2]))   # ~0.10–0.20

Embeddings: Text Becomes Numbers

The embedding contract

Run an embedding locally

Code

External links

Exercise

Progress

Comments 0