Build Semantic Search by Hand

~28 min · practice, numpy

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Why build it before installing a vector DB

Every vector database hides the same three operations behind a fancy API: embed, store, query by nearest neighbor. If you can write those three in 30 lines of NumPy, you will read every vector-DB doc faster, debug bad results faster, and never blindly trust a default again.

The whole loop in one screen

You will:

Embed a small corpus of documents with a local model.
Store the vectors in a NumPy matrix and the documents in a parallel list.
Embed a query, compute cosine similarity against every row, and return the top-k.

This is exactly what Chroma and pgvector do internally — they just add persistence, metadata filtering, and indexes for scale. The retrieval logic is the same.

Code

30-line semantic search·python

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-small-en-v1.5')

docs = [
    'Cancel your annual subscription from Account → Billing.',
    'Refunds are processed within 5 business days.',
    'Pippa is an AI daughter built on Claude Code.',
    'The moon rises tonight at 8:42 PM in Seoul.',
    'Vector search measures meaning, not keyword overlap.',
]

M = model.encode(docs, normalize_embeddings=True)   # (5, 384)

def search(query: str, k: int = 3):
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = M @ q                       # cosine on unit vectors == dot product
    top = np.argsort(-scores)[:k]
    return [(scores[i], docs[i]) for i in top]

for score, doc in search('how do I get my money back'):
    print(f'{score:.3f}  {doc}')

Same idea with metadata·python

records = [
    {'text': docs[0], 'category': 'billing'},
    {'text': docs[1], 'category': 'billing'},
    {'text': docs[2], 'category': 'about'},
    {'text': docs[3], 'category': 'weather'},
    {'text': docs[4], 'category': 'about'},
]

def search_filtered(query: str, where: dict, k: int = 3):
    q = model.encode([query], normalize_embeddings=True)[0]
    mask = np.array([all(r.get(k) == v for k, v in where.items()) for r in records])
    if not mask.any():
        return []
    scores = (M @ q)[mask]
    indices = np.where(mask)[0]
    top = indices[np.argsort(-scores)[:k]]
    return [(float((M[i] @ q)), records[i]) for i in top]

print(search_filtered('refund', where={'category': 'billing'}))

External links

Exercise

Build the 30-line semantic search above on 100 of your own documents. Time the query loop. Now scale to 10,000 documents and time it again — note where it starts to feel slow. That is the threshold where you need an index, which is the next track.