Embeddings
How each token becomes a list of numbers, so that meaning turns into distance.
Read first:Tokenisation
Step 1 of 6· Hand-set 12-number vectors; the cosine and analogy maths is real, the 2D map is an approximate projection
After tokenisation, each token id picks a row from a large learned table. That row, a list of hundreds or thousands of numbers, is the token's embedding. Training nudges the numbers so that words used in similar contexts end up with similar lists. Treat each list as a point in space and “cat” lands near “dog”, “Paris” near “Rome”. Closeness is measured with cosine similarity: how nearly two lists point the same way. Directions can carry meaning too: in early word vectors, king − man + woman landed close to queen. Search by meaning works the same way: embed the question, then find the nearest stored items.
Why it matters for your product
Embeddings let a product search by meaning rather than exact words, so a support assistant can find the refunds policy when a customer writes “money back”. The choice of embedding model largely decides what search can find, so test it on your own questions before trusting a leaderboard, and embed queries and documents with the same model. Vectors also take space: a million documents at 1,024 four-byte numbers each is about 4 GB before any index.
For engineersShow the maths
cos(a, b) = (a · b) / (‖a‖ × ‖b‖)
Multiply the two vectors position by position and add up, then divide by both lengths, so only the direction counts, not the size.
Worked example: a = (1, 2), b = (2, 3): a · b = 8, ‖a‖ = 2.24, ‖b‖ = 3.61, so cos = 8 / 8.06 = 0.99.
answer = the word w, other than a, b and c, with the highest cos(x_b − x_a + x_c, x_w)
To solve “a is to b as c is to ?”, scale each vector to length 1, add the a-to-b offset to c and take the nearest word, skipping the three question words.
Worked example: With this page's vectors, king − man + woman is closest to queen, at 0.97.
Embeddings absorb the associations, including the biases, of the text they were trained on, and the famous analogies are fragile: the standard method is not allowed to return any of the input words, which can make results look cleaner than they are. Raw vectors are not automatically good for search either: compared by cosine, raw BERT outputs scored below simple averaged GloVe vectors until Sentence-BERT trained them for similarity.
The words you will hear
Embedding
Primary sources
- 01A Neural Probabilistic Language ModelBengio et al. · 2003
- 02Efficient Estimation of Word Representations in Vector SpaceMikolov et al. · 2013
- 03Linguistic Regularities in Continuous Space Word RepresentationsMikolov et al. · 2013
- 04Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksReimers, Gurevych · 2019
- 05Fair is Better than Sensational: Man is to Doctor as Woman is to DoctorNissim et al. · 2019
Connected ideas
Retrieval and search
HNSW vector search
How a stack of sparse-to-dense graphs finds the nearest vectors without checking them all.
ExploreRetrieval and search
Retrieval-augmented generation (RAG)
How a model answers from your documents, and shows you exactly where each claim came from.
ExploreInside the transformer
Self-attention
How every word looks at every other word to work out what it means here.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.