How Embeddings Work, Counted by Hand

An embedding turns a word into a list of numbers, arranged so that words with similar meanings end up near each other. Every semantic search engine, every recommendation feed and every RAG pipeline rests on that one trick.

The part almost nobody explains is where the numbers come from. They are not assigned by a linguist, and they are not arbitrary. They are, at bottom, counted.

Two words that are alike without ever meeting

The tool below reads your text and gives every word a vector: one dimension per other word, each holding how often the two appear near each other. Nothing more.

Look at the default corpus. cat and dog score 0.955 — and they never appear next to each other. Not once. There is no sentence anywhere in that text linking them.

They score alike because they are surrounded by the same words: sat, mat, ate, food, slept, fire. Meanwhile cat and engine score 0.068, because an engine keeps entirely different company.

That is the whole idea, and it has a name: the distributional hypothesis. A word is characterised by the words it turns up near. You never have to tell the machine that a cat resembles a dog — it falls out of counting.

The same thing in Python

import re
from collections import Counter, defaultdict
from math import sqrt

text = """
the cat sat on the mat. the dog sat on the mat.
the cat ate the food. the dog ate the food.
the cat slept near the fire. the dog slept near the fire.
the engine burned the fuel. the engine turned the wheel.
"""

STOP = {"the", "a", "an", "and", "or", "of", "to", "in", "on", "at", "for", "with", "is", "it"}
words = [w for w in re.findall(r"[a-z']+", text.lower()) if len(w) > 1 and w not in STOP]

WINDOW = 2
vectors = defaultdict(Counter)
for i, word in enumerate(words):
    lo, hi = max(0, i - WINDOW), min(len(words), i + WINDOW + 1)
    for j in range(lo, hi):
        if i != j:
            vectors[word][words[j]] += 1


def cosine(a, b):
    shared = set(a) & set(b)
    dot = sum(a[k] * b[k] for k in shared)
    if not dot:
        return 0.0
    return dot / (sqrt(sum(v * v for v in a.values())) * sqrt(sum(v * v for v in b.values())))


print("cat's vector:", dict(vectors["cat"].most_common(4)))
print("dog's vector:", dict(vectors["dog"].most_common(4)))
print()
print("cat next to dog ever? ", "dog" in vectors["cat"])
print()
for other in ("dog", "engine", "fire"):
    print(f"cosine(cat, {other:<7}) = {cosine(vectors['cat'], vectors[other]):.3f}")

cat's vector: {'sat': 2, 'mat': 2, 'ate': 2, 'food': 2}
dog's vector: {'sat': 2, 'mat': 2, 'ate': 2, 'food': 2}

cat next to dog ever?  False

cosine(cat, dog    ) = 0.955
cosine(cat, engine ) = 0.068
cosine(cat, fire   ) = 0.295

Twenty lines, two dictionaries, one square root. That is a word embedding.

Why angle, and not distance

Every embedding library compares vectors with cosine similarity — the angle between them — rather than straight-line distance. This is presented as a convention far more often than it is explained.

The reason is that raw distance measures the wrong thing. Take a word that appears twice in your corpus and the same word in a corpus ten times larger. Every count in its vector is roughly ten times bigger. The word means the same thing; the vector is ten times longer.

• Cosine ignores length entirely. Multiply every number in a vector by ten and the angle does not move — the similarity stays exactly 1.0.
• Euclidean distance is dominated by that scaling. The same pair now looks far apart.

Frequency is not meaning. Cosine throws away the magnitude, which here is mostly noise, and keeps the direction, which is the part carrying the meaning.

You can make the two measures visibly disagree. Paste this into the tool and select cat:

the cat sat on the mat. the cat sat on the mat. the cat sat on the mat.
the cat sat on the mat. the cat sat on the mat. the cat sat on the mat.
the dog sat on the mat.
the beam sat on the mat.
the beam held the roof. the beam held the roof. the beam held the roof.
the beam braced the roof. the beam braced the roof.

cat now appears six times and dog once, in identical surroundings. The two rankings split:

By angle, beam is fourth. By distance it is second — ahead of sat and mat — purely because its vector happens to be a similar size to cat’s. Nothing about a structural beam is more cat-like than a mat is.

And dog scores a cosine of exactly 1.000: its direction is identical, because its contexts are identical. Its distance is 12.7 all the same, entirely because it was mentioned once instead of six times. Distance is reporting how often, and we asked about meaning.

From counting to learning

What the demo builds is a sparse count vector: one dimension per vocabulary word, nearly all zeros. Real systems use dense learned vectors — a few hundred dimensions, none of which correspond to a particular word. But the raw material is the same.

GloVe, from Stanford, is the clearest bridge. It is trained on a global word-word co-occurrence matrix — the very thing the tool above builds — and its stated objective is to learn vectors whose dot product equals the logarithm of the words’ co-occurrence probability. In other words: count first, then compress. Its published vectors run from 50 to 300 dimensions, over vocabularies from 400,000 to 2.2 million words.

word2vec (Mikolov and colleagues, 2013) takes the other route — rather than tabulating co-occurrence, it trains a model to predict which words appear in a context, and keeps the weights. Different mechanism, same underlying bet: a word’s company determines its representation.

Modern sentence and document embeddings go further again, using transformers so that the same word gets a different vector in different sentences. The distributional bet still holds. It is just made with far more machinery.

What this demo simplifies

• Sparse, not dense. One dimension per word instead of a few hundred learned ones. Real vectors are compressed; these are not.
• No weighting. Real pipelines apply PPMI or a similar correction so that common words do not dominate. This uses raw counts, with a small stopword list as a blunt substitute.
• No learning. Nothing is optimised here. GloVe and word2vec both fit parameters against an objective; this only counts.
• Tiny corpus. Eight sentences produce neighbours that are suggestive, not reliable. Real training corpora run to billions of words, and that scale is doing much of the work.

None of these change the mechanism. They change how well it performs.

Where to go next

The code above is two dictionaries and some arithmetic — Counter, defaultdict, and set intersection. If those are unfamiliar, the linked lessons cover them, and they run in the browser exactly like the snippet on this page.

Once embeddings make sense, retrieval does too: searching by meaning rather than keyword is this same cosine comparison, run against a stored index instead of four sentences.