{"id":"quantization","level":"software","name":"Quantization and numerical representation","shortName":"Quantization","role":"Represents model values with reduced precision to lower storage and arithmetic costs.","description":"Quantization maps values into a limited set of representable levels. For a simple symmetric integer example, divide by a scale, round, and clamp into the available range; dequantization reconstructs an approximation using that scale. Other formats, including low-bit floating point and block-scaled schemes, use different representations. The atlas shows a continuous value distribution landing on discrete levels and makes clipping visible at the extremes. Fewer bits can reduce model storage and data movement, but metadata, scales, unquantized tensors, and execution support affect actual savings. Accuracy must be measured on the intended tasks. Quantization is an engineering tradeoff in representation, not a guarantee that every smaller format preserves the same model behavior.","specs":[{"label":"Teaching equation","value":"q = clamp(round(x / s))","context":"Symmetric integer example with positive scale s."},{"label":"INT8 representation","value":"8 bits per stored value","context":"Actual tensors can also need scales and other metadata."},{"label":"Precision choices","value":"Weights, activations, and accumulation","context":"These need not all use the same format."}],"interfaces":[{"name":"Higher-precision tensors and scales","direction":"in","signal":"Values and representation parameters"},{"name":"Quantized representation","direction":"out","signal":"Reduced-precision values for supported execution"}],"tradeoff":"Fewer bits reduce some memory and compute costs. Rounding, clipping, and unsupported paths can erase the expected benefit.","whyAI":"Lower precision can make models cheaper to serve or train. Quality and performance must be evaluated together on the real workload.","related":["compiler","kernel","serving","hbm-stack"],"sources":[{"title":"NVIDIA TensorRT — Quantize Operator","url":"https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/Quantize.html"},{"title":"NVIDIA TensorRT — Quantization Schemes","url":"https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/quantized-types-schemes.html"}],"scale":"Numerical value-space model; no physical scale","kind":"quantization","mechanism":["Choose a representable range and scaling rule for the selected format.","Map values to allowed levels, including rounding and any clipping.","Execute with supported arithmetic and evaluate output quality and achieved performance."],"misconception":"An eight-bit weight format does not imply that all activations, accumulators, metadata, or runtime memory use eight bits.","check":{"question":"What happens to a value outside the chosen quantization range?","options":["It becomes infinitely precise","It always expands the hardware word size","It may be clipped to the representable boundary"],"answer":2,"explanation":"Clipping loses information at the extremes, so scale selection and task-level evaluation matter."},"branch":"Model deployment","subbranch":"Numerical representation","science":[{"title":"Unclipped rounding-error bound","equation":"x̂ = s × round(x/s); |x − x̂| ≤ s/2","explanation":"Uniform nearest rounding limits each unclipped scalar’s reconstruction error to half a quantization step. Clipping and downstream error propagation are separate and can be much more consequential.","assumptions":"Positive constant scale, nearest rounding, and values inside the representable range; does not bound end-to-end model error or describe every floating-point/block format."}],"history":[],"prerequisites":["mac"],"learningObjective":"Calculate a scalar reconstruction and apply the unclipped nearest-rounding error bound.","engineeringExample":"Synthetic symmetric quantizer with step s = 0.25 maps x = 0.62 to round(0.62/0.25) = 2 and reconstructs 0.5. Absolute error is 0.12, below s/2 = 0.125. A value outside the allowed integer range can be clipped and need not satisfy this bound.","evidenceNotes":"The linked quantization references supply the representation context. This example assumes positive fixed scale and nearest rounding without clipping; it does not bound model accuracy or describe every low-bit floating-point format.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}