Quantization stores a model’s numbers in fewer bits. A model trained with 16-bit or 32-bit weights is converted to 8-bit, 4-bit or even lower precision, so it needs a fraction of the memory and usually runs faster. The price is a small loss of accuracy, which can be negligible or serious depending on the method, the model and the task. Quantization is the main reason large models can run on a single GPU, a laptop or a phone.
How Quantization Works
A neural network is mostly large tables of numbers. Quantization maps each number from a fine scale onto a coarse one. With 4 bits there are only 16 possible values per weight, so the method chooses a scale and offset for each small group of weights that keeps the rounding error as low as possible.
What gets quantized:
- Weights only. The most common choice for language models. Memory drops sharply and quality holds up well.
- Weights and activations. Also compresses the values flowing through the network, which speeds up computation on hardware that supports low-precision maths.
- The KV cache. The memory a model keeps for the text it has already read. Compressing it helps with long context windows.
When it happens:
- Post-training quantization (PTQ) converts a finished model, often using a small calibration dataset. It takes minutes to hours and needs no retraining. GPTQ is a well-known example: its authors quantized a 175-billion-parameter model to 3 or 4 bits in about four GPU hours with negligible accuracy loss on their benchmarks, enough to run it on a single GPU.
- Quantization-aware training (QAT) simulates low precision during training so the model learns to cope with it. Better quality at very low bit widths, but it needs a training run.
Quantization also combines with fine-tuning. QLoRA trains small adapter layers on top of a frozen 4-bit model, which made it possible to fine-tune a 65-billion-parameter model on one 48 GB GPU.
Choosing a Precision
| Precision | Typical use | What to expect |
|---|---|---|
| 16-bit | Reference quality, training | Baseline |
| 8-bit (integer or floating point) | Production serving on modern GPUs | Usually very close to baseline |
| 4-bit | Self-hosting large models on fewer GPUs, laptops | Small loss, task-dependent |
| 3-bit and below | Extreme memory limits, edge devices | Visible loss unless the method is specialised |
These are rules of thumb, not guarantees. The only reliable number is the one you measure on your own evaluation set.
When Not to Quantize
- When you call a hosted API. The vendor handles serving precision and you cannot change it.
- When a task is already at the edge of what the model can do. Long multi-step reasoning, maths and code tend to degrade first.
- When the bottleneck is elsewhere. If latency comes from retrieval or network calls, a smaller model file will not help.
Common Failure Modes
- Testing on the wrong data. A quantized model can match the original on a public benchmark and still slip on your domain terms or language.
- Mismatched calibration data. PTQ calibrated on generic English text can hurt a model used for another language or for code.
- Hardware assumptions. A format that is fast on one GPU family can be slow or unsupported on another, or on CPU.
- Stacking losses. Quantizing a model that was already distilled to a small size compounds the quality drop.
Related Terms
Quantization, model distillation and pruning are the three common ways to make a model cheaper to run. Quantization is what makes small language models and open-weight models practical on modest hardware.
In Practice
Quantization is measured, never assumed to be free. The quantized model is compared with the original on an evaluation set of real inputs and on latency, memory and cost, and the format, such as 8-bit or 4-bit weights, is chosen for the hardware it will run on. Tasks that need exact output, such as extraction, code or arithmetic, get the closest look, since small losses tend to show there first.
Quantization is a routine step in our edge AI development, private LLM and data residency and MLOps services work.