Embeddings as Features

~26 min · embeddings, representation

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

What embeddings give you

An embedding is a fixed-length numeric vector that summarizes a complex object: a sentence, an image, a product, a user session. Once embedded, classical ML models can ride on top: kNN for retrieval, logistic regression for classification, lightgbm for tabular fusion. Embeddings turn unstructured inputs into tabular features.

Where embeddings come from

Pretrained models — sentence-transformers, OpenAI text-embedding-3, image encoders (CLIP).
Fine-tuned models — start from a pretrained base, fine-tune on your domain.
Self-trained — train an autoencoder or contrastive model on your own data.

Production discipline

Embeddings are model artifacts. Pin the model version, the tokenizer version, and the embedding dimension in your feature schema. A silent embedding-model upgrade is the worst kind of drift, because everything downstream still works but means something subtly different.

Code

Sentence embeddings for tabular fusion·python

from sentence_transformers import SentenceTransformer
import numpy as np

encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeds = encoder.encode(df["description"].tolist(), show_progress_bar=True)
embeds = np.asarray(embeds, dtype="float32")
X_full = np.hstack([X_tabular, embeds])

Vector search on stored embeddings·python

import numpy as np
from sklearn.preprocessing import normalize

matrix = normalize(embeds)  # cosine sim = dot product after L2-normalize
query = normalize(encoder.encode(["premium subscription churn"]))
sims = matrix @ query.T
top = np.argsort(-sims.ravel())[:10]

External links

Exercise

Pick one free-text column in your dataset. Encode it with sentence-transformers MiniLM. Concatenate to your tabular features. Compare CV score with and without embeddings. Quantify the lift before deciding to ship the encoder.