{
  "schemaVersion": 1,
  "id": "atlas-lesson-quantization",
  "title": "Quantization and numerical representation",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Represents model values with reduced precision to lower storage and arithmetic costs.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/quantization/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model deployment",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The linked quantization references supply the representation context. This example assumes positive fixed scale and nearest rounding without clipping; it does not bound model accuracy or describe every low-bit floating-point format.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/quantization-7283db1a755017ec.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#quantization",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "249952a95cafe3ae475fb1de972d3ab38b508da44e02e7c50d0dc3f5ea439363",
  "contentLength": 5334,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-quantization.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-quantization.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Quantization and numerical representation",
      "start": 33,
      "end": 388
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 388,
      "end": 1432
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1432,
      "end": 1460
    },
    {
      "id": "section-4",
      "title": "Unclipped rounding-error bound",
      "start": 1460,
      "end": 2346
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2346,
      "end": 2613
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2613,
      "end": 3024
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3024,
      "end": 3355
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3355,
      "end": 3535
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3535,
      "end": 3564
    },
    {
      "id": "section-10",
      "title": "What happens to a value outside the chosen quantization range?",
      "start": 3564,
      "end": 3933
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3933,
      "end": 4334
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4334,
      "end": 4812
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4812,
      "end": 4835
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4835,
      "end": 4944
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4944,
      "end": 5334
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#quantization"
    },
    {
      "title": "NVIDIA TensorRT — Quantize Operator",
      "url": "https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/Quantize.html"
    },
    {
      "title": "NVIDIA TensorRT — Quantization Schemes",
      "url": "https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/quantized-types-schemes.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/quantization-7283db1a755017ec.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Multiply–accumulate and fused arithmetic",
      "url": "https://semiaifoundry.com/learn/atlas/mac/"
    },
    {
      "title": "Compiler, intermediate code, and device executable",
      "url": "https://semiaifoundry.com/learn/atlas/compiler/"
    },
    {
      "title": "GPU kernel and thread hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/kernel/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "High-bandwidth memory stack",
      "url": "https://semiaifoundry.com/learn/atlas/hbm-stack/"
    }
  ],
  "recordHash": "9b3aa2525d77e90ddfc2ab4f91b90fe5f99fb689c3887bc2f147d2c31febe9f1",
  "contentFormat": "text/markdown",
  "content": "Model deployment · Atlas v1.3.0\n\n# Quantization and numerical representation\n\nRepresents model values with reduced precision to lower storage and arithmetic costs.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#quantization)Numerical value-space model; no physical scale\n\nLearning objectiveCalculate a scalar reconstruction and apply the unclipped nearest-rounding error bound.\n\n## The mechanism\n\nQuantization maps values into a limited set of representable levels. For a simple symmetric integer example, divide by a scale, round, and clamp into the available range; dequantization reconstructs an approximation using that scale. Other formats, including low-bit floating point and block-scaled schemes, use different representations. The atlas shows a continuous value distribution landing on discrete levels and makes clipping visible at the extremes. Fewer bits can reduce model storage and data movement, but metadata, scales, unquantized tensors, and execution support affect actual savings. Accuracy must be measured on the intended tasks. Quantization is an engineering tradeoff in representation, not a guarantee that every smaller format preserves the same model behavior.\n\n1.  Choose a representable range and scaling rule for the selected format.\n2.  Map values to allowed levels, including rounding and any clipping.\n3.  Execute with supported arithmetic and evaluate output quality and achieved performance.\n\n## Science and assumptions\n\n### Unclipped rounding-error bound\n\n$$\n\\hat x=s\\operatorname{round}(x/s),\\qquad|x-\\hat x|\\le\\frac{s}{2}\n$$\n\nReconstruction multiplies the nearest integer to x divided by step size by that step size. Without clipping, absolute scalar error is at most half a step.\n\nSymbols and units\n\nx\n\nOriginal scalar value\n\ndeclared value units\n\nx̂\n\nReconstructed quantized scalar\n\nsame units as x\n\ns\n\nPositive quantization step\n\nsame units as x\n\nround\n\nNearest-integer rounding operator\n\ndimensionless integer output\n\nUniform nearest rounding limits each unclipped scalar’s reconstruction error to half a quantization step. Clipping and downstream error propagation are separate and can be much more consequential.\n\nAssumptions Positive constant scale, nearest rounding, and values inside the representable range; does not bound end-to-end model error or describe every floating-point/block format.\n\n## Engineering example\n\nSynthetic symmetric quantizer with step s = 0.25 maps x = 0.62 to round(0.62/0.25) = 2 and reconstructs 0.5. Absolute error is 0.12, below s/2 = 0.125. A value outside the allowed integer range can be clipped and need not satisfy this bound.\n\n## Distinctions and tradeoffs\n\nAn eight-bit weight format does not imply that all activations, accumulators, metadata, or runtime memory use eight bits.\n\nFewer bits reduce some memory and compute costs. Rounding, clipping, and unsupported paths can erase the expected benefit.\n\nLower precision can make models cheaper to serve or train. Quality and performance must be evaluated together on the real workload.\n\n## Specifications and interfaces\n\nTeaching equation\n\nq = clamp(round(x / s))\n\nSymmetric integer example with positive scale s.\n\nINT8 representation\n\n8 bits per stored value\n\nActual tensors can also need scales and other metadata.\n\nPrecision choices\n\nWeights, activations, and accumulation\n\nThese need not all use the same format.\n\n### Interfaces\n\nHigher-precision tensors and scales Input\n\nValues and representation parameters\n\nQuantized representation Output\n\nReduced-precision values for supported execution\n\n## Check your understanding\n\n### What happens to a value outside the chosen quantization range?\n\n1.  It becomes infinitely precise\n2.  It always expands the hardware word size\n3.  It may be clipped to the representable boundary\n\nAnswer and reasoning\n\nIt may be clipped to the representable boundary\n\nClipping loses information at the extremes, so scale selection and task-level evaluation matter.\n\n## Evidence and scope\n\nThe linked quantization references supply the representation context. This example assumes positive fixed scale and nearest rounding without clipping; it does not bound model accuracy or describe every low-bit floating-point format.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [NVIDIA TensorRT — Quantize Operator](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/Quantize.html)\n-   [NVIDIA TensorRT — Quantization Schemes](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/quantized-types-schemes.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/quantization-7283db1a755017ec.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n\n### Related lessons\n\n-   [Compiler, intermediate code, and device executable](https://semiaifoundry.com/learn/atlas/compiler/)\n-   [GPU kernel and thread hierarchy](https://semiaifoundry.com/learn/atlas/kernel/)\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/)"
}