The architecture story: from the transformer to reasoning and agents

The transformer paper was submitted on 12 June 2017. Its contribution was subtractive: it removed recurrence and convolution and relied on attention alone. The reported results were 28.4 BLEU on WMT14 English-German and a new single-model state of the art of 41.8 BLEU on English-French after 3.5 days of training on eight GPUs.

The important property was not translation quality. It was that the number of sequential operations became constant rather than growing with sequence position, so all positions could be computed in parallel. That is what made the later scale-ups affordable, and it is the reason every step in this story rests on the same block.

Two follow-ups turned the block into a recipe. OpenAI's 2018 paper showed that generatively pre-training a decoder-only transformer on unlabelled text and then fine-tuning per task improved the state of the art on 9 of 12 benchmarks, beating architectures built for each task. GPT-3, submitted 28 May 2020 at 175 billion parameters, pushed that far enough that tasks could be specified purely in the prompt, with no gradient updates at all.

Kaplan et al., submitted 23 January 2020, found that loss scales as a power law with model size, dataset size and compute across seven orders of magnitude, and that width and depth choices matter little within a wide range. The practical reading at the time was: make the model very large, train on relatively modest data, stop early.

Chinchilla, submitted 29 March 2022, corrected that. For a fixed compute budget, parameters and training tokens should be scaled roughly equally. A 70B model trained on 4x more data outperformed the 280B Gopher at the same compute, reaching 67.5% on MMLU, over seven points above Gopher.

Commercially, this is the correction with the clearest consequence for a product budget. A smaller model trained longer costs less to serve at equal quality, and serving cost is what a product pays every day. Every "small but well-trained" open model since follows this logic.

InstructGPT, submitted 4 March 2022, added two stages after pre-training: supervised fine-tuning on labeller demonstrations, then reinforcement learning from human rankings. Labellers preferred outputs from the 1.3B InstructGPT model to those of the 175B GPT-3, despite roughly 100x fewer parameters.

Constitutional AI, submitted 15 December 2022, replaced most of the human harm-labelling. The model critiques and revises its own outputs against a written list of principles, then a preference model trained on AI judgements drives the reinforcement learning stage, an approach usually written as RLAIF.

For builders, the consequence is structural. Instruction-following, formatting reliability and refusal behaviour are properties of a post-training stage, which is why they can change noticeably between model versions from the same vendor without a new base model. Behaviour you depend on should be pinned to a model version and covered by tests.

Switch Transformer, submitted 11 January 2021, simplified mixture-of-experts routing to a single expert per token, reported up to 7x faster pre-training than T5-Base and T5-Large at equal compute, and pre-trained models up to a trillion parameters. Mixtral 8x7B, submitted 8 January 2024, brought the pattern into open weights: 8 feedforward experts per layer with 2 selected per token, 47B total but 13B active, matching or beating Llama 2 70B and GPT-3.5 on tested benchmarks under Apache 2.0.

DeepSeek-V3, whose technical report was submitted 27 December 2024, was the reference point for scale at the time: 671B total and 37B active, with Multi-head Latent Attention, auxiliary-loss-free load balancing and multi-token prediction, pre-trained on 14.8 trillion tokens in 2.788 million H800 GPU-hours without loss spikes or rollbacks.

The practical consequence is a change in how you read a model card. Total parameters set your memory footprint; active parameters set your compute per token. A 671B model can be cheaper per token than a dense 70B one, and a 47B model can need more GPU memory than its speed suggests.

Rotary Position Embedding, submitted 20 April 2021, encodes absolute position as a rotation so that self-attention depends on relative position, giving sequence-length flexibility and a decaying dependency with distance. It became the default positional scheme in open models. Ring Attention, submitted 3 October 2023, distributes long sequences across devices and fully overlaps key-value block communication with blockwise attention, so context length scales with device count without approximation.

Products followed. Google announced Gemini 1.5 Pro on 15 February 2024 with a 1-million-token context window in limited preview and 128,000 tokens as the standard tier; the technical report of 8 March 2024 reported near-perfect retrieval, above 99%, up to at least 10 million tokens across text, video and audio. Claude Sonnet 4 added 1M-token support on 12 August 2025 in public beta.

The pricing is the part product teams miss. Above 200K tokens, Claude Sonnet 4 input is $6 per million tokens against $3 at or below, and output is $22.50 against $15. Long context removes an engineering constraint and adds a budget line.

Three ideas dominate serving cost. Grouped-Query Attention, submitted 22 May 2023, uses an intermediate number of key-value heads between multi-head and multi-query attention, and uses the same uptraining recipe that converts existing multi-head checkpoints with about 5% of the original pre-training compute. DeepSeek-V2, submitted 7 May 2024, introduced Multi-head Latent Attention, compressing the KV cache into a latent vector: 93.3% less KV cache and 5.76x higher maximum generation throughput compared with DeepSeek 67B.

Speculative decoding, submitted 30 November 2022, is the odd one out because it changes nothing about the model's answers. A small approximation model drafts tokens that the large model verifies in parallel, giving a 2x to 3x speedup on T5-XXL with outputs identical in distribution to standard decoding.

Together these explain why long-context and high-concurrency workloads became affordable without new hardware. They also give you a category of optimisation that carries no quality risk, which is rare enough to be worth naming explicitly when planning a latency budget.

Snell et al., submitted 6 August 2024, showed that adapting inference compute per prompt difficulty, using verifier-guided search and iterative revision, improves efficiency by more than 4x over best-of-N sampling, and can let a smaller model outperform one 14x larger on questions where the base model has a non-trivial chance of success. The finding is that where you spend compute matters as much as how much you spend.

OpenAI released o1-preview and o1-mini on 12 September 2024, trained with large-scale reinforcement learning to reason through a chain of thought, with performance improving both with more RL at training time and with more thinking time at test time. The o1 system card, published 21 December 2024, describes the same training approach and its use for reasoning about safety policies in context.

DeepSeek-R1, submitted 22 January 2025, showed that pure reinforcement learning with verifiable rewards and no supervised reasoning demonstrations produces emergent self-reflection, verification and strategy adaptation, and that a cold-start supervised phase plus distillation transfers those behaviours into smaller models.

The consequence for products is a new cost axis. An answer now has a variable token count and a variable latency that depend on how hard the model finds the question. That has to be budgeted, capped and monitored, the same way you would treat a query that sometimes triggers a full table scan.

On 13 June 2023 OpenAI introduced function calling for gpt-4-0613 and gpt-3.5-turbo-0613: developers describe a function signature and the model chooses to emit a JSON object of arguments conforming to it. That turned "the model decides to call a tool" from prompt scaffolding into an API primitive with a schema.

GPT-4o, announced 13 May 2024, merged voice, text and vision into a single model rather than the previous pipeline of separate models, which OpenAI's CTO said had led to longer response times and higher compute cost. Removing the speech-to-text, LLM and text-to-speech hops is what cuts the round-trip latency of a voice interaction, which OpenAI's CTO gave as a reason for merging the pipeline into one model.

Anthropic released computer use in public beta on 22 October 2024 with an upgraded Claude 3.5 Sonnet, which scored 14.9% on OSWorld screenshot-only, and 22.0% with more steps, against a prior best of 7.8%. That was a large relative jump and a low absolute number, and it was labelled experimental. It is a good reminder to read agent benchmarks as both at once.

Anthropic open-sourced the Model Context Protocol on 25 November 2024 as an open standard replacing fragmented per-source integrations with one protocol, with Block and Apollo as early adopters and Zed, Replit, Codeium and Sourcegraph building support. The specification revision dated 2025-06-18 defines stateful JSON-RPC 2.0 connections between hosts, clients and servers: servers expose Resources, Prompts and Tools, while clients may offer Sampling, Roots and Elicitation, with elicitation added in that revision alongside the existing sampling and roots features.

That specification also states that hosts must obtain explicit user consent before invoking any tool. This is worth reading as an architectural requirement rather than a policy note: consent has to live in the host application, which means your product owns the approval surface, not the model vendor.

Google announced the Agent2Agent protocol on 9 April 2025 with over 50 technology partners, built on HTTP, Server-Sent Events and JSON-RPC, and positioned as complementary to MCP. The division of labour is that MCP supplies tools and context to one agent, while A2A coordinates between agents.

So the stack has stabilised into four layers: a post-trained, often sparse model; a structured tool-calling interface; MCP for supplying tools and context; and optionally A2A for cross-agent delegation. That stability is the practical news.