In-context learning
Teaching a model a task with examples in the prompt, while its weights stay exactly as they were.
Read first:Next-token prediction
Step 1 of 6· Illustrative probabilities for a base model; the accuracy figures quoted on steps 4 and 6 are from the cited papers
In-context learning is teaching by example without any training. You put a few worked examples in the prompt, then a new case, and the model continues the pattern. Its weights never change: the examples shape its next-token probabilities only while they sit in the prompt. More examples usually help, but which ones you pick, and in what order, can swing the answer. Chain-of-thought prompting extends the idea: examples that show their working lead the model to write out steps too, and each written step gives it more context for the next, which helps on multi-step problems.
Why it matters for your product
Few-shot prompting is often the fastest way to adapt a model: no training run, no new model to host, and a change is one edit away. The costs are real, though. Examples add tokens to every call, and results can shift with example choice and order, so treat examples as part of the product: version them and test them against a fixed evaluation set that includes awkward cases. Asking for step-by-step reasoning can raise accuracy on multi-step tasks, at the price of longer, slower answers.
For engineersShow the maths
answer ~ P_θ( · | instruction, x₁ → y₁, …, x_k → y_k, x ) with θ unchanged
The model is the same function with the same weights θ; only the text it conditions on changes. In-context learning is conditioning, not training.
Worked example: In the demo, P(billing) goes from 22% to 48% to 86% as the prompt gains 0, 1 and 3 examples, while the number of weights changed stays at 0.
input tokens per call = instruction + k × tokens per example + question
Every example is re-sent and re-read on every call, so the prompt cost grows in step with k.
Worked example: A 30-token instruction, 5 examples of 40 tokens and a 25-token question make 255 input tokens per call: 255,000 a day at 1,000 calls.
Examples mostly show the model the format and the set of possible answers: one study found that even random labels in the examples barely hurt accuracy, so do not expect a few examples to teach a mapping the model does not already know. Written reasoning is not guaranteed to be the real reason for an answer; chain-of-thought explanations can misrepresent why a model chose what it did.
The words you will hear
In-context learning
Primary sources
- 01Language Models are Few-Shot LearnersBrown et al. · 2020
- 02Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsWei et al. · 2022
- 03Large Language Models are Zero-Shot ReasonersKojima et al. · 2022
- 04Calibrate Before Use: Improving Few-Shot Performance of Language ModelsZhao et al. · 2021
- 05Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Min et al. · 2022
- 06Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingTurpin et al. · 2023
Connected ideas
Training and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreInterpretability and safety
Prompt injection
When text the model reads on your behalf starts giving it orders.
ExploreAgents and protocols
The agent loop
How a model works through a task: think, use a tool, read the result, and go round again.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.