Quantisation
Storing a model's numbers with fewer bits, and keeping the answers nearly the same.
Read first:Neural networks
Step 1 of 7· Illustrative weights; the rounding maths is real, and the OPT-175B results are from the GPTQ paper
Quantisation stores a model's numbers with fewer bits. Weights usually take 16 bits each. Quantisation maps them onto a small set of evenly spaced levels (at most 16 with 4 bits) plus a scale factor that turns the levels back into real values. Each weight is rounded to its nearest level, which adds a small error but cuts memory by two to four times. Because writing each new token is mostly limited by how fast the weights can be read from memory, smaller weights can also mean faster answers. The hard part is a few unusually large values, which careful methods handle so that quality barely moves.
Why it matters for your product
Quantisation often decides whether a model fits on one GPU or needs several. A 4-bit copy needs a quarter of the memory of the 16-bit original and can generate tokens faster, but only if the serving stack has fast kernels for that format, and the quality loss varies by model, task and method. Before switching, run your own evaluation set on the quantised model rather than trusting a headline benchmark.
For engineersShow the maths
s = max|w| / (2ᵇ⁻¹ − 1), q = round(w / s), ŵ = s · q
Pick a scale so the largest weight lands on the top level, store each weight as the nearest whole number q, and multiply back by s when the weight is used.
Worked example: 4 bits gives levels −7 to +7. If max|w| = 0.7 then s = 0.1, and a weight of 0.234 is stored as q = 2 and read back as 0.2: an error of 0.034.
memory = parameters × bits ÷ 8 (bytes)
Memory for the weights is simply how many there are times how many bits each one takes.
Worked example: 175 billion × 16 ÷ 8 = 350 GB. At 4 bits it is 87.5 GB: a quarter of the size.
extra bits per weight = bits per scale ÷ group size
Per-group scales are not free: every group stores its own scale, spread across the weights in the group.
Worked example: A 16-bit scale per 128 weights adds 16 ÷ 128 = 0.125 bits, so 4-bit weights cost about 4.1 bits each. GPTQ reports about 0.15 extra bits for its groups of 128.
At 3 bits and below, plain rounding breaks down and even careful methods lose some accuracy, by amounts that vary with the model and the task. Speed-ups need hardware kernels for the format: without them a quantised model saves memory but may not run any faster.
The words you will hear
Bit width
Primary sources
- 01Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only InferenceJacob et al. · 2017
- 02LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleDettmers et al. · 2022
- 03GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersFrantar et al. · 2022
- 04AWQ: Activation-aware Weight Quantization for LLM Compression and AccelerationLin et al. · 2023
Connected ideas
Speed and cost
Mixture of experts
A model with many specialist blocks, where each token only visits two of them.
ExploreTraining and alignment
Distillation
Training a small model to imitate a large one, including what the large one nearly said.
ExploreTraining and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreGenerating text
KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.