The mechanism
Quantization maps values into a limited set of representable levels. For a simple symmetric integer example, divide by a scale, round, and clamp into the available range; dequantization reconstructs an approximation using that scale. Other formats, including low-bit floating point and block-scaled schemes, use different representations. The atlas shows a continuous value distribution landing on discrete levels and makes clipping visible at the extremes. Fewer bits can reduce model storage and data movement, but metadata, scales, unquantized tensors, and execution support affect actual savings. Accuracy must be measured on the intended tasks. Quantization is an engineering tradeoff in representation, not a guarantee that every smaller format preserves the same model behavior.
- Choose a representable range and scaling rule for the selected format.
- Map values to allowed levels, including rounding and any clipping.
- Execute with supported arithmetic and evaluate output quality and achieved performance.
Science and assumptions
Unclipped rounding-error bound
Reconstruction multiplies the nearest integer to x divided by step size by that step size. Without clipping, absolute scalar error is at most half a step.
Symbols and units
- x
- Original scalar value
declared value units
- x̂
- Reconstructed quantized scalar
same units as x
- s
- Positive quantization step
same units as x
- round
- Nearest-integer rounding operator
dimensionless integer output
Uniform nearest rounding limits each unclipped scalar’s reconstruction error to half a quantization step. Clipping and downstream error propagation are separate and can be much more consequential.
Assumptions Positive constant scale, nearest rounding, and values inside the representable range; does not bound end-to-end model error or describe every floating-point/block format.
Engineering example
Synthetic symmetric quantizer with step s = 0.25 maps x = 0.62 to round(0.62/0.25) = 2 and reconstructs 0.5. Absolute error is 0.12, below s/2 = 0.125. A value outside the allowed integer range can be clipped and need not satisfy this bound.
Distinctions and tradeoffs
An eight-bit weight format does not imply that all activations, accumulators, metadata, or runtime memory use eight bits.
Fewer bits reduce some memory and compute costs. Rounding, clipping, and unsupported paths can erase the expected benefit.
Lower precision can make models cheaper to serve or train. Quality and performance must be evaluated together on the real workload.
Specifications and interfaces
- Teaching equation
- q = clamp(round(x / s))
Symmetric integer example with positive scale s.
- INT8 representation
- 8 bits per stored value
Actual tensors can also need scales and other metadata.
- Precision choices
- Weights, activations, and accumulation
These need not all use the same format.
Interfaces
- Higher-precision tensors and scales Input
- Values and representation parameters
- Quantized representation Output
- Reduced-precision values for supported execution
Check your understanding
What happens to a value outside the chosen quantization range?
- It becomes infinitely precise
- It always expands the hardware word size
- It may be clipped to the representable boundary
Answer and reasoning
It may be clipped to the representable boundary
Clipping loses information at the extremes, so scale selection and task-level evaluation matter.
Evidence and scope
The linked quantization references supply the representation context. This example assumes positive fixed scale and nearest rounding without clipping; it does not bound model accuracy or describe every low-bit floating-point format.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.