How a big model fits in fewer bits

A language model is a very long list of numbers, its weights. How many bytes each one takes decides whether the model fits in your machine and how quickly its numbers can be read from memory. Quantization stores each number in fewer bits, by rounding it to the nearest of a short list of allowed values.

The first sum is the same for every model: the memory for the weights is how many there are times the bytes each takes, 2 for a 16-bit number, half a byte for a 4-bit one.

By the end you will know:

• what “4-bit” means: a grid of allowed values and a scale that stretches it;
• why one big value ruins the rounding for everyone, and how groups of weights fix it at a price;
• how to tell whether a model fits your memory, and what that check leaves out.

To keep a weight in 4 bits you cannot keep it exactly. Take the largest weight in a group, ignoring its sign, lay a ruler from minus that value to plus that value with evenly spaced marks, and store each weight as the number of its nearest mark, a small integer. The group stores one scale: the gap between marks.

scale s = largest absolute weight ÷ (2^bits−1 − 1) stored integer q = round(w ÷ s) value used w′ = q × s

With 4 bits the integer runs from −7 to 7: 15 marks, leaving one of the 16 codes unused so the ruler is symmetric. Drag the weight, then pull the bits slider from 8 to 2 to watch the allowed values thin out.

This figure needs JavaScript. It shows a weight of 0.37 on a ruler from −1 to 1 and the marks a given number of bits allows. With 4 bits there are 15 marks, 1/7 apart, so 0.37 is stored as the integer 3 and read back as 0.429, an error of 0.059, which can never be more than half a step (0.071). With 8 bits there are 255 marks and the error is under 0.001. With 2 bits there are only three marks, −1, 0 and 1, and 0.37 becomes 0: the weight disappears.

The error is the gap between the ring and the mark, never more than half a gap. Each bit removed about doubles the gap and the worst-case error (from 3 bits to 2 it triples). At 2 bits it lands on 0: it vanishes.

So the model keeps integers, not the real numbers? An integer per weight and one scale per group, and it computes with their product. The real number is gone; only the nearest mark is left.

The ruler’s ends come from the largest weight, and that is the weak spot. One value far bigger than the rest sets the gap between marks, and the ordinary values, all between the two marks around zero, round to zero. The LLM.int8() paper (Dettmers and colleagues, 2022) found a version of this in large language models: a few hidden dimensions with values up to about 20 times larger than the rest, which its authors keep in 16 bits. In the paper they are in hidden states rather than weights; the row below shares only the arithmetic.

The row is invented: 32 weights, one about fourteen times larger than the biggest of the others, all rounded to 4 bits. Choose how many weights share a scale, then switch the outlier off.

This figure needs JavaScript. It shows 32 invented weights, one of them about fourteen times larger than the rest, rounded to 4 bits. With one scale for all 32 the big value sets a step of 0.857 and 31 of the weights round to zero (mean squared error 0.0275, 4.5 bits per weight with a 16-bit scale). Groups of 16 give 0.0101 at 5 bits per weight, groups of 8 give 0.0028 at 6 and groups of 4 give 0.0024 at 8. Without the big value one scale for all 32 gives 0.00027.

With one scale for all 32, the outlier sets the gap at 0.857 and 31 weights round to zero. In groups of 8 only the weights that share its group suffer, and the error falls from 0.0275 to 0.0028, about a tenth. Without the outlier the choice hardly matters (0.00027).

The price is in the last readout. Each group stores a scale, here 16 bits, which adds 16 ÷ g bits to every weight of a group of g: half a bit at 32, four at 4, and a scale for every single weight would make a 4-bit weight cost 20 bits, five times as much. The llama.cpp Q4_0 block has this layout, 32 four-bit weights and one 16-bit scale: 4.5 bits per weight.

So smaller groups are always better? No: better for the error, worse for the bill. On this row groups of 8 at 4 bits (6 bits per weight) only match one scale at 6 bits (6.5), and one scale at 8 bits (8.5, error 0.00015) is 16 times better than groups of 4 at 4 bits (8).

A row has one weight about fourteen times larger than the rest, rounded to 4 bits with a single scale. What happens to the ordinary weights?

1. Nothing: each weight is rounded on its own.
2. Most round to zero: the big value sets the gap between marks.
3. They are kept in 16 bits automatically, because the scale notices the odd value.

Show answer

Most round to zero: in the figure, one scale for all 32 wipes out 31.

Not A: the marks are shared, so the biggest value sets the step for all of them.

Not C: plain rounding does not look for outliers; LLM.int8() adds a 16-bit path on purpose.

A language model is too big to play with here, so this figure uses a toy: two inputs, two hidden layers of 16 units, one output, trained to tell two interleaved moons apart. Only its weights are rounded; biases and arithmetic stay at full precision. Per layer shares one scale across a layer; per neuron gives every neuron its own, the grouping idea again.

Drag the bits down from 32. The dashed line is the original network’s boundary between the moons, the solid line the rounded one’s.

This figure needs JavaScript. It shows a tiny network that separates two interleaved moons, with its weights rounded to fewer bits. At full precision it gets 197 of 200 test points right. With one scale per layer it still gets 197 at 8 bits, 190 at 4 bits (95.0 percent, 4.16 bits per weight with the scales), 126 at 3 bits (63.0) and 100 at 2 bits, where it gives one answer everywhere. With one scale per neuron it gets 195 at 4 bits (5.74 bits per weight) and 183 at 3 (4.74): more scales, more bits. It is a toy, not a language model.

At 8 bits the lines sit on each other and the score holds: 197 of 200 points, 98.5 percent. At 4 bits per layer the boundary wobbles (95.0); at 3 it breaks (63.0); at 2 the network answers one class everywhere. One scale per neuron keeps 97.5 at 4 bits and 91.5 at 3, but it stores 33 scales instead of 3, and the readout counts them: 3 bits per neuron cost 4.74 bits per weight, 4 bits per layer 4.16, and the layer version scores 95.0 against 91.5. At the same true cost the finer scales bought nothing here. We trained sixteen such networks and show the seventh; its 4-bit score (95.0) is above their average (89.9), and on average 3 bits per neuron (90.2) and 4 bits per layer (89.9) tie. A toy shows the shape of the trade, not where the drop falls for a real model; the GPTQ authors found larger models generally easier to quantize.

In the tiny network, 3 bits with one scale per neuron score far higher than 3 bits with one scale per layer. What explains it?

1. Per neuron rounds with a more careful rule, so each weight lands closer to its mark.
2. It uses fewer bits in total, because one scale per neuron is smaller than one per layer.
3. A neuron with small weights gets finer marks, at the price of more scales to store.

Show answer

A neuron’s scale comes from its own largest weight, so small-weight neurons get finer marks. The price: 33 scales instead of 3, about 1.6 more bits per weight.

Not A: the rule is the same, nearest mark; only the size of the marks changes.

Not B: per neuron stores 33 scales instead of 3, so it costs more bits, not fewer.

Now memory. Pick a size and a card, then drag the bits. The bar is the memory the weights take, with a 16-bit scale per 32 weights, so “4 bits” costs 4.5. The hatched part is what is left for everything else: the KV cache, which holds what the model has already read and grows with the text, the activations (its working numbers) and the software. Those depend on the model, the program and the text, so the figure does not guess: it shows only what is left. Sizes are round numbers, not particular models.

This figure needs JavaScript. It compares the memory the weights of a model take with the memory of a card, in units of 2^30 bytes, counting half a bit per weight for the scales. A model of 13 billion weights takes 24.2 GB at 16 bits, which does not fit a card of 16 GB (8.2 over), 12.9 GB at 8 bits, which fits with 3.1 left, and 6.8 GB at 4 bits. The KV cache, the activations and the software need memory too, and the figure does not guess how much.

13 billion numbers at 16 bits are 24.2 GB, 8.2 too many for a 16 GB card. At 8 bits they take 12.9 and leave 3.1; at 4 bits, 6.8. A 70-billion model fits a 24 GB card only at 2 bits, with 3.6 to spare. “Fits” says only that the weights fit. The biggest model that fits at 4 bits is often, not always, the best pick: across more than 35,000 experiments on four model families (Dettmers and Zettlemoyer, 2022), 4-bit weights were almost always the best trade of accuracy against total bits.

This is the sum on the model card of Hati Glint SQL: 9 billion bf16 weights of 2 bytes need about 16.8 GB of GPU memory (1 GB here is 2³⁰ bytes). That is a fact about the 16-bit weights; this post claims nothing about any smaller version of that model.

A card has 24 GB and the weights of a model come to 22 GB. What does the bar tell you?

1. The weights fit, but only about 2 GB is left for the KV cache and the rest.
2. It fits, and nothing else needs memory.
3. It does not fit: a model should never take more than half of the card’s memory.

Show answer

The weights fit, with about 2 GB left. Whether that covers the rest depends on the program and the text.

Not B: the KV cache, the activations and the software need memory too.

Not C: there is no such rule; what counts is what is left.

Rounding to the nearest mark ignores what the weights are for. Three ideas do better.

Choose better values. GPTQ (Frantar and colleagues, 2022) rounds one layer at a time and picks values that keep its output close to the original on a small sample of text. In its tests plain rounding to 3 bits collapsed on large models and GPTQ held up.

Protect what matters. AWQ (Lin and colleagues, 2023) finds the weights that matter most by watching the numbers that flow through the model, not by their size. At 3 bits on a 6.7-billion model, plain rounding scored a perplexity of 23.5 (how badly the model predicts text; lower is better; 10.9 at 16 bits). Keeping 1 percent of the channels in 16 bits cut it to 11.4 when they were chosen that way, and to 22.4 when chosen by weight size.

Train with rounding in mind. Jacob and colleagues (2017) simulate the rounding while training. QLoRA (Dettmers and colleagues, 2023) instead freezes a finished model in 4 bits and trains small add-on layers on it, enough to fine-tune a 65-billion model on one 48 GB card.

All of this rounds weights. The numbers flowing through the model are harder to round, and that is where the outliers of LLM.int8() live.

If the arithmetic is still 16-bit, where does the saving come from? Memory traffic. Writing text is limited by how fast the weights are read, the fact the speculative decoding post builds on, so fewer bytes help (the GPTQ authors credit their speed-ups to less memory movement).

The quality you lose depends on the model, the method and the task; this post gives no such number for any RuneLab model. Another route to a smaller model is to train a small one to imitate a big one: see the model distillation post.

The source of every outside number and claim, as it stood on 2026-10-15 (the llama.cpp file is a living page).

1. Benoit Jacob and colleagues, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference” (2017): the scale and integer scheme, training with simulated quantization.
2. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale” (2022): outlier features, up to about 20 times larger.
3. Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers” (2022): the 3-bit collapse of plain rounding, memory movement, larger models.
4. Ji Lin and colleagues, “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration” (2023): the 1 percent experiment (section 3.1, Table 1).
5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs” (2023): the 65-billion model on 48 GB.
6. Tim Dettmers and Luke Zettlemoyer, “The case for 4-bit precision: k-bit Inference Scaling Laws” (2022): more than 35,000 experiments, 4 bits almost universally best.
7. The llama.cpp project, ggml-common.h: the block layouts of Q4_0 (32 weights and a 16-bit scale in 18 bytes) and Q8_0 (34 bytes).
8. RuneLab, the Hati Glint SQL model card: the bf16 weights need about 16.8 GB of GPU memory.

• Quantization rounds each weight to the nearest of a few allowed values; every bit removed about doubles the gap.
• One big value stretches a shared ruler, so scales are kept per group, and “4-bit” means about 4.5 bits per weight once they are counted.
• A toy shows the shape of the trade; measure the quality on your own task.
• The weights fitting is the first test, not the last: the KV cache and the activations need memory too.