Tokenisation
How text is cut into the numbered pieces a model actually reads, and why everything is counted in them.
Step 1 of 6· A real byte-pair encoder on a five-word toy corpus; the window sizes are illustrative
A language model never sees letters or words. It sees tokens: chunks of text, each with an id number. The chunks are learned before training, often with byte-pair encoding. Start from single bytes, find the pair that appears most often in a large sample of text, glue it into a new token, and repeat many thousands of times. Common words end up as one token; rare words are spelled from several pieces, so no text is ever unwritable. Everything a model handles is counted in tokens: how much it can read at once and what you pay. The same sentence can need far more tokens in one language than another.
Why it matters for your product
Tokens are the unit of cost, speed and memory: providers bill per token, longer inputs take longer to process, and the context window caps what a model can hold at once. So measure prompts in tokens, not characters; budget the window explicitly and reserve room for the answer; and test the languages and formats your users actually send, because the same meaning can take several times more tokens in some languages than in English.
For engineersShow the maths
vocabulary size = base symbols + number of merges
Every merge adds exactly one new token, and the number of merges is the only setting the algorithm has.
Worked example: This page's tokeniser starts from 256 bytes and learns 10 merges, so its vocabulary is 256 + 10 = 266 tokens.
system + documents + history + answer budget ≤ context window
Everything sent to the model, plus the room reserved for its reply, has to fit inside one budget.
Worked example: In the visual: 768 + 3 × 1,024 + 1,024 + 1,536 = 6,400 tokens in an 8,192-token window, leaving 1,792 to spare.
The tokeniser is fixed before training, so text it splits badly stays expensive: the same content can take up to 15 times more tokens in some languages than in others. A larger window holds more, but accuracy and recall degrade as it fills, so more context is not automatically better.
The words you will hear
Token
Primary sources
- 01Neural Machine Translation of Rare Words with Subword UnitsSennrich et al. · 2015
- 02Language Models are Unsupervised Multitask LearnersRadford et al. · 2019
- 03SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text ProcessingKudo, Richardson · 2018
- 04Language Model Tokenizers Introduce Unfairness Between LanguagesPetrov et al. · 2023
- 05Context windowsAnthropic · 2026
Connected ideas
Language as numbers
Embeddings
How each token becomes a list of numbers, so that meaning turns into distance.
ExploreGenerating text
Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
ExploreRetrieval and search
Retrieval-augmented generation (RAG)
How a model answers from your documents, and shows you exactly where each claim came from.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.