Double Espresso: What Is Quantisation?

What Is Quantisation?

Download a language model to run on your own computer, and you may be offered several versions with labels such as sixteen bit, eight bit, or four bit. They can share the same architecture and the same number of parameters, yet require very different amounts of memory. The main reason is quantisation: representing the model's numbers with fewer possible values.

A trained model contains billions of numerical weights. During a calculation it also creates temporary numbers called activations. Computers never store these with infinite precision. They use formats with a fixed number of bits. More bits can represent values with finer detail or a wider range, but they also take more space and move more data through the machine.

Quantisation maps many finely represented values onto a smaller set of levels. Imagine a shop that records every ingredient weight to the nearest gram, then switches to measuring in ten-gram steps. Eleven grams and fourteen grams might both be recorded as ten. The ledger becomes simpler, but small distinctions disappear. Model quantisation is more sophisticated: different groups of numbers can receive their own scales, important outliers can be treated specially, and the available levels need not resemble round decimal steps. The basic bargain remains. Fewer representable values cost less space, while rounding changes the calculation.

The storage arithmetic can be striking. If each weight changes from sixteen bits to four, the raw weight data needs one quarter as many bits. That can turn a model that does not fit in a graphics card's memory into one that does. It can also reduce the traffic between memory and the processor, which is often a major bottleneck during generation.

But one quarter is not a promise about the whole application. The quantised model needs scales and other metadata. Some layers may stay at higher precision. Activations, the growing context cache, temporary workspaces, and the software itself also consume memory. A four-bit weight file can therefore be roughly a quarter of the original weight file without making every part of the running system four times smaller.

Smaller does not automatically mean faster either. Speed depends on whether the hardware and its low-level software routines can calculate efficiently in the chosen format. A processor may have excellent support for eight-bit arithmetic but need awkward conversion work for a particular four-bit scheme. On another device, the smaller memory traffic may dominate and produce a large gain. A quantisation label without a named runtime and machine is only part of a performance claim.

Most practical quantisation is lossy. The new model is not bit-for-bit identical to the original because many weights have been rounded. Well-designed methods try to place that error where it matters least. Some examine representative inputs to find sensitive values. Some quantise weights while leaving activations at higher precision. Others reshape the problem so that both weights and activations can use fewer bits. There is no single technique called “the four-bit version”.

Nor can we inspect one rounded weight and translate its error into one misspelled word. A model's output comes from many layers of interacting calculations. Tiny numerical changes may cancel, accumulate, or alter which of two almost equally likely tokens is selected. That distributed behaviour is why engineers measure the completed model after conversion. File integrity can show that we downloaded the intended artifact. Evaluation can then sample how it behaves on the cases we care about.

Well-designed conversions can show only a small change on selected evaluations. But “small on these tests” does not mean “impossible to notice”. Effects can differ across languages, rare knowledge, mathematical tasks, long contexts, instruction following, and particular prompts. A lower-precision model can occasionally choose a different first token, after which an entire generated answer diverges. The sensible comparison is the task you actually care about, not merely the download size.

It also helps to separate quantisation from three neighbouring ideas. Pruning removes weights or connections judged unnecessary. Distillation trains a new, usually smaller model to imitate a larger one. Quantisation normally keeps the same parameter structure and changes how its numbers are represented. A model with thirty billion four-bit parameters still has thirty billion parameters; each weight is simply stored with less precision.

Quantisation can happen after training, starting with an existing high-precision model, or the lower-precision constraint can be anticipated during training. The details matter to engineers, but the user-facing question is simpler: which artifact, on which hardware, preserves enough quality for the intended job at an acceptable memory, speed, and energy cost?

This also explains why a format name is not a quality grade. Two four-bit conversions can group weights differently, choose different scales, use different calibration material, or preserve different components at higher precision. One may suit a laptop processor; another may be designed around a data-centre accelerator. “Four bit” tells us the broad compression level, not every decision behind it.

So a quantised model is best understood as a compressed numerical rendition, not a miniature photocopy guaranteed to behave identically. It can bring capable models onto cheaper or local hardware and make each generated token less demanding. The trade is not fewer learned connections. It is fewer bits with which to express the values of those connections, managed carefully enough that the useful behaviour survives.