Semantic Arithmetic: king − man + woman ≈ queen

~10 min · semantics, intuition

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The famous result from word2vec (Mikolov et al., 2013) is the cleanest existence proof that embeddings encode structured meaning rather than random codes. The vector for "king" minus "man" plus "woman" is closest, in the trained embedding space, to the vector for "queen."

The intuition: certain directions in the embedding space correspond to consistent semantic relationships. Subtracting "man" from "king" isolates a vector that encodes royalty + masculinity. Adding "woman" replaces the masculinity component with femininity. The result lands near "queen."

Other classic examples: Paris − France + Italy ≈ Rome; walking − walk + swim ≈ swimming; good − bad + evil ≈ malevolent. These are not perfect; they hold approximately for a subset of relationships, and modern Transformer embeddings (which include all of context, not just the lookup) capture much richer structure than simple word-level analogies suggest.

Code

Run the analogy yourself·python

import numpy as np

def analogy(a, b, c, embeddings, vocab, k=5):
    # a is to b as c is to ? -> find token closest to (b - a + c)
    va, vb, vc = embeddings[vocab[a]], embeddings[vocab[b]], embeddings[vocab[c]]
    target = vb - va + vc
    sims = embeddings @ target / (
        np.linalg.norm(embeddings, axis=1) * np.linalg.norm(target) + 1e-9
    )
    # Exclude the input tokens themselves so we don't 'cheat'
    for tok in (a, b, c):
        sims[vocab[tok]] = -1
    top = sims.argsort()[::-1][:k]
    inv = {v: k for k, v in vocab.items()}
    return [inv[i] for i in top]

# analogy("man", "king", "woman", emb, vocab) -> ['queen', ...]

External links

Exercise

Using GPT-2's embedding matrix, run analogy(a, b, c) for 10 pairs you care about. Half should be 'easy' (capital-country, gender, tense), half should be hard (occupation→tool, abstract concepts, multi-word). Tabulate the top-5 results. Where does the analogy fail and why?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousEmbedding Dimension (d_model): 768 to 12,288 Next →The Position Problem: Attention Is Permutation-Invariant

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.