What Is LLM Quantization? Bits, Memory, Speed, and Quality

Precision determines how values are represented

Neural-network parameters and intermediate values are numbers. A full-precision or half-precision format allocates more bits to represent each value than a lower-precision integer or floating-point format. Quantization maps values from a higher-precision representation into a smaller set of representable values.

For model weights, the raw storage relationship is approximately:

weight bytes = parameter count × bits per stored parameter ÷ 8

A dense model with 10 billion parameters would have an ideal raw weight payload of about 20 GB at 16 bits per parameter, 10 GB at 8 bits, or 5 GB at 4 bits. These are decimal approximations for the weight payload only. A real checkpoint and running process also need scales, metadata, unquantized modules, runtime buffers, KV cache, and other overhead.

Quantization is therefore a way to reduce one major part of the memory budget, not a guarantee that the entire application shrinks by exactly the same ratio.

What can be quantized

Different methods quantize different values:

• Weight-only quantization stores model weights at lower precision while computation may use another data type.
• Weight-and-activation quantization lowers precision for both weights and intermediate activations.
• KV-cache quantization reduces the memory used for attention state during inference.
• Optimizer-state quantization reduces training or fine-tuning memory rather than ordinary inference memory.

A model label such as “4-bit” is incomplete without the method, group size, scale representation, compute type, and runtime. Two 4-bit checkpoints can differ in quality, speed, and memory.

Post-training quantization and quantization-aware training

Post-training quantization (PTQ) converts an already trained model. Many downloadable LLM formats use PTQ because it avoids full retraining. Calibration data may be used to choose scales or identify sensitive weights.

Quantization-aware training (QAT) simulates or incorporates quantization effects during training so the model can adapt. It can preserve quality in some settings but requires a training process and suitable data.

Methods such as GPTQ and activation-aware weight quantization are examples of approaches designed to reduce LLM weight precision while controlling error. Libraries and runtimes also implement numeric formats such as int8, FP8, and NF4, plus packed checkpoint representations and hardware-specific variants.

Why quantization can reduce memory

Lower-bit weights require fewer bytes to store and transfer. This can allow a model to fit on a smaller accelerator, leave more room for KV cache and concurrency, or reduce the number of devices needed for one replica.

The theoretical weight saving is straightforward, but deployed memory includes:

• Quantized weight data
• Per-group scales and possible zero points
• Layers retained at higher precision
• Dequantization or compute buffers
• Runtime workspaces
• KV cache
• Activations
• Framework and allocator overhead

Use the LLM memory requirements guide to calculate the complete budget rather than multiplying parameters by one number.

Why quantization may improve or hurt speed

Quantization can improve speed by reducing memory bandwidth and using faster low-precision hardware operations. It can also make a model slower when the runtime repeatedly dequantizes values, uses an inefficient kernel, transfers data between devices, or lacks native support for the format.

Performance depends on:

• GPU, CPU, or accelerator architecture
• Serving runtime and kernel implementation
• Batch size and sequence length
• Whether the workload is compute-bound or memory-bound
• Quantization format and group size
• Tensor parallelism and device placement
• Prefill versus decoding behavior

Do not infer throughput from file size. Benchmark the exact checkpoint, runtime, hardware, input length, output length, and concurrency.

Quality loss is task-dependent

Quantization introduces approximation error. Whether that error is acceptable depends on the model, method, bit width, calibration data, and task. A checkpoint can perform well on broad benchmarks but regress on a narrow production workflow.

Evaluate at least:

• Task success rate
• Structured-output validity
• Retrieval or classification accuracy
• Code execution or test pass rate
• Long-context behavior
• Tool-call correctness
• Safety and refusal behavior relevant to the product
• Retry rate

Retry rate belongs in the evaluation because a cheaper first pass that fails more often can increase total cost.

Quantization is not pruning or distillation

Pruning removes weights, neurons, attention heads, or other structure. Distillation trains a smaller student model to reproduce useful behavior from a larger teacher. Quantization changes numeric representation.

These techniques can be combined, but they change the model in different ways. A quantized 70-billion-parameter model still has approximately the same parameter count even though each stored parameter uses fewer bits.

Quantization and fine-tuning

Low-bit base models can be used with parameter-efficient fine-tuning. QLoRA, for example, keeps a quantized frozen base model and trains added low-rank adapters. This lowers the memory required for adaptation compared with full-parameter fine-tuning.

That does not mean ordinary 4-bit weights are updated directly in every setup. The training library, quantization method, and adapter design determine which parameters are trainable.

For inference, adapter weights and runtime support add their own memory and operational considerations.

How to select a quantized checkpoint

Use a repeatable sequence:

1. Establish the unquantized model’s quality and performance baseline.
2. List hardware and runtime formats that are actually supported.
3. Calculate weight memory plus context and concurrency memory.
4. Select two or three quantization candidates.
5. Run the same task-level evaluation on each candidate.
6. Benchmark latency and throughput with realistic sequence lengths.
7. Measure peak memory, not only checkpoint size.
8. Include retries and failures in effective cost.
9. Preserve the exact model, method, runtime, and configuration in results.

The open-model token cost calculator can convert measured throughput and hourly compute price into a unit cost. It should receive benchmarked values, not generic model-family claims.

Frequently asked questions

Does 4-bit quantization make a model four times smaller than FP16?

The ideal raw weight payload is roughly one quarter as large, but real memory includes scales, metadata, higher-precision modules, runtime buffers, KV cache, and allocator overhead. Measure the deployed process.

Is an 8-bit model always faster than a 16-bit model?

No. Speed depends on hardware support, kernels, batch size, and bottlenecks. Lower memory use does not automatically produce higher throughput.

Does quantization change the model’s answers?

It can. Quantization approximates values and may change token probabilities or task behavior. Validate the exact checkpoint on production-representative tests.

Can quantization reduce KV-cache memory?

Some runtimes support KV-cache quantization, but weight quantization alone does not automatically change the KV-cache format. Treat the two configurations separately.

Sources

• Hugging Face Transformers: bitsandbytes quantization
• GPTQ: Accurate post-training quantization for generative pretrained transformers
• AWQ: Activation-aware weight quantization for LLM compression and acceleration
• QLoRA: Efficient finetuning of quantized LLMs