Mixture of experts
A model with many specialist blocks, where each token only visits two of them.
Read first:Feed-forward networkTransformer architecture
Step 1 of 6· Illustrative router scores; the softmax, top-2 and capacity maths is real, and the Mixtral figures are published
A mixture-of-experts model replaces the single feed-forward block in some or all layers with several parallel blocks, called experts, plus a small router. For every token, the router scores all the experts, sends the token to only the best few (often two out of eight) and blends their answers. So the model stores far more parameters than it uses on any one token: Mixtral 8x7B holds 47 billion but uses about 13 billion per token. That buys the capacity of a big model at closer to the running cost of a small one. The catch is balance: training has to stop the router sending everything to a few favourite experts.
Why it matters for your product
Mixture of experts is why some very large open models are affordable to run: you pay compute for the active parameters but memory for all of them. That changes hardware sizing, because the whole model must still fit in GPU memory or be split across GPUs, and moving tokens between GPUs adds communication cost. When comparing models, keep active and total parameters in separate columns: one predicts speed, the other memory.
For engineersShow the maths
y = Σᵢ∈T gᵢ · Eᵢ(x), g = softmax(Top-k(x · W_g))
Score all experts with one small matrix W_g, keep the k largest scores (the set T), softmax just those into weights g, and add up the chosen experts' outputs by weight.
Worked example: Top-2 scores of 2.0 and 1.5: softmax gives e^0.5 / (e^0.5 + 1) ≈ 0.62, so the weights are 62% and 38%.
active share of expert weights = k / N
With N experts and top-k routing, each token touches k of the N experts, so compute on the expert layers falls to k/N of a model that ran them all.
Worked example: k = 2, N = 8 gives 25%. Mixtral uses 13B of its 47B parameters per token, about 28%: a little above 25% because attention and embeddings are shared by every token.
loss_aux = α · N · Σᵢ fᵢ · Pᵢ
fᵢ is the share of tokens whose first choice is expert i and Pᵢ is the average router probability for expert i. The sum is smallest when both are even, at 1/N each; Switch Transformer used α = 0.01.
Worked example: Four experts, perfectly even: 4 × 4 × (¼ × ¼) = 1. Every token to expert 1 with probability 0.9: 4 × (1 × 0.9) = 3.6.
Mixture of experts saves compute, not memory: every expert must be loaded, so a 47-billion-parameter model needs the memory of one even though it runs more like a 13-billion one. Routing tokens to experts on different GPUs adds communication, and tokens that overflow an expert's capacity skip it.
The words you will hear
Expert
Primary sources
- 01Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerShazeer et al. · 2017
- 02Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityFedus et al. · 2021
- 03Mixtral of ExpertsJiang et al. · 2024
Connected ideas
Inside the transformer
Feed-forward network
The per-token layer that widens each vector, filters it and writes back what it recalls.
ExploreSpeed and cost
Quantisation
Storing a model's numbers with fewer bits, and keeping the answers nearly the same.
ExploreTraining and alignment
Pretraining and scaling laws
Why loss falls predictably with compute, and how to split a budget between model size and data.
ExploreSpeed and cost
Batching and PagedAttention
Keeping a GPU busy with many conversations at once, and storing their memory in pages.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.