Encoders vs decoders
The same transformer blocks, trained either to understand a whole text or to write the next word.
Read first:Self-attention
Step 1 of 6· Illustrative sentence and guesses; the mask patterns and BERT's 15% masking rule are real
Transformers come in two main kinds built from the same parts. An encoder, such as BERT, lets every word look at the words on both sides, and learns by filling in words that were blanked out. That makes it good at understanding a whole piece of text: sorting emails, or turning documents into vectors for search. A decoder, such as GPT, lets each word look only at the words before it, and learns by predicting the next word. That is the skill needed to write text one word at a time. Some models, such as T5, use both: an encoder reads the input and a decoder writes the output.
Why it matters for your product
Choosing the family is a cost and quality decision. Ranking documents, routing tickets or spotting duplicates rarely needs a large generative model: encoders turn text into embeddings or scores. In the Sentence-BERT paper, precomputed embeddings cut finding the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds. Writing, summarising and conversation need a decoder. Retrieval systems often use both: encoders to find and rerank passages, then a decoder to write the answer from them.
For engineersShow the maths
Encoder mask: Mᵢⱼ = 0 for all i, j. Decoder mask: Mᵢⱼ = 0 if j ≤ i, −∞ if j > i
The mask is added to the attention scores before softmax. A score of −∞ becomes a weight of exactly zero, so a blocked token contributes nothing.
Worked example: For 6 tokens an encoder uses all 36 scores; a decoder keeps 21 of them (6 × 7 ÷ 2).
MLM loss = − Σ over chosen i of log P(xᵢ | the corrupted sequence)
Only the chosen positions are scored, whether they were masked, swapped or left unchanged, and each prediction may use context from both sides.
Worked example: A 512-token sequence with 15% of positions chosen gives about 77 predictions to learn from.
Next-token loss = − Σᵢ log P(xᵢ | x₁, …, xᵢ₋₁)
Every position predicts the token that follows it, using only the tokens before it.
Worked example: The same 512-token sequence gives 511 predictions, one for every position that has a next token.
The split is a tendency, not a law: the original GPT decoder was itself fine-tuned for classification and question answering. Encoders, trained to fill blanks with both sides visible, are not set up to write text one token at a time.
The words you will hear
Encoder
Primary sources
- 01BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingDevlin et al. · 2018
- 02Improving Language Understanding by Generative Pre-TrainingRadford et al. · 2018
- 03Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerRaffel et al. · 2019
- 04Attention Is All You NeedVaswani et al. · 2017
- 05Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksReimers, Gurevych · 2019
- 06Passage Re-ranking with BERTNogueira, Cho · 2019
Connected ideas
Generating text
Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
ExploreLanguage as numbers
Embeddings
How each token becomes a list of numbers, so that meaning turns into distance.
ExploreRetrieval and search
Hybrid search and reranking
Search by exact words and by meaning at once, merge the two lists, then let a slower model reread the best few.
ExploreRetrieval and search
Retrieval-augmented generation (RAG)
How a model answers from your documents, and shows you exactly where each claim came from.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.