Retrieval and search

Retrieval-augmented generation (RAG)

How a model answers from your documents, and shows you exactly where each claim came from.

Basics · 7 steps

Read first:Embeddings HNSW vector search Hybrid search and reranking

Step 1 of 7· Invented documents, a toy concept-count embedding, reranker and tokeniser; the chunking, cosine and budget arithmetic is real

use the steps or arrow keys

Loading visual…

in one minute

A language model only knows what was in its training data, and it can fill gaps with fluent guesses. Retrieval-augmented generation, or RAG, hands it the relevant parts of your own documents at the moment it answers. Ahead of time, the documents are cleaned, cut into chunks and turned into vectors stored in an index. When a question arrives, the system finds the chunks closest to it, keeps the best few and pastes them into the prompt with numbered labels. The model is told to answer only from those passages and cite them, so each claim can be checked. Change a document and the next answer changes, with no retraining.

Why it matters for your product

RAG lets a model work from an organisation's own policies, contracts and tickets without retraining, and citations let staff check an answer in seconds. Many bad answers are retrieval failures, so first measure whether the right chunk reaches the prompt. Tune chunk size on your own documents, keep passages few and put the strongest first: models use the start and end of a long prompt better than the middle. For a collection under about 200,000 tokens, Anthropic suggests skipping retrieval and including all of it.

For engineersShow the maths

cos(q, c) = (q · c) ÷ (|q| × |c|)

The similarity between the question's vector q and a chunk's vector c: their dot product, divided by both lengths so a chunk does not score higher just for being long.

Worked example: In the toy model the question touches five concepts once each, and the 9 pm sentence has four of them, one three times: 6 ÷ (√5 × √12) = 0.77.

Σ tokens(passageᵢ) ≤ passage budget

Every kept passage spends part of a fixed budget; a passage that would go over it is left out, together with everything it says.

Worked example: With chunks of up to 60 words, the first two passages cost 63 + 32 = 95 of 100 tokens, so the 26-token passage holding the 30-day deadline does not fit.

p(y | x) ≈ Σ over top-k passages z of p(z | x) × p(y | x, z)

The original RAG-Sequence model retrieved the top k passages for input x, generated the answer y from each one, and weighted each by how relevant the retriever judged that passage. Most systems today simply paste the passages into one prompt instead.

Worked example: With k = 5, the probability of an answer is a weighted average of five probabilities, one computed from each passage.

where it stops working

RAG is only as good as its retrieval: if the passage holding the answer is not found, or is cut in half, the model cannot use it and may answer from memory instead. Citations are not proof either: on one long-form question set, even the best models Gao et al. tested lacked full citation support for about half of their answers.

Key terms