{
  "schemaVersion": 1,
  "id": "atlas-lesson-floating-point-reduction-order",
  "title": "Floating-point reductions and compiler permissions",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Distinguish algebraic equivalence, numerical error and bitwise reproducibility when compilation or collectives regroup arithmetic.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/floating-point-reduction-order/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Executable software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The NVIDIA guide supports finite-precision nonassociativity, and LLVM specifies the cited optimization permissions. The exact operands, intermediate values and 0.01 tolerance are authored. Tests use binary32 conversions locally and do not constitute a GPU, compiler or multi-rank experiment.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/floating-point-reduction-order-b8b43418bbe78f69.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#floating-point-reduction-order",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "fa34872623b3da0576a25593def8a324e7d8f106846277f70fb017fe2ab33514",
  "contentLength": 8775,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-floating-point-reduction-order.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-floating-point-reduction-order.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Floating-point reductions and compiler permissions",
      "start": 36,
      "end": 563
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 563,
      "end": 2430
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2430,
      "end": 2458
    },
    {
      "id": "section-4",
      "title": "Rounding makes grouping observable",
      "start": 2458,
      "end": 3170
    },
    {
      "id": "section-5",
      "title": "Spacing at the chosen magnitude",
      "start": 3170,
      "end": 3900
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 3900,
      "end": 4739
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 4739,
      "end": 5327
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5327,
      "end": 5724
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5724,
      "end": 5943
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5943,
      "end": 5972
    },
    {
      "id": "section-11",
      "title": "With binary32 ties-to-even rounding, what is fl(fl(2²⁴+1)−2²⁴)?",
      "start": 5972,
      "end": 6220
    },
    {
      "id": "section-12",
      "title": "Why does grouping 2²⁴+(1−2²⁴) instead give 1?",
      "start": 6220,
      "end": 6589
    },
    {
      "id": "section-13",
      "title": "Which evidence shows a result is accurate relative to a stated reference?",
      "start": 6589,
      "end": 6927
    },
    {
      "id": "section-14",
      "title": "Evidence and scope",
      "start": 6927,
      "end": 7395
    },
    {
      "id": "section-15",
      "title": "Sources",
      "start": 7395,
      "end": 7890
    },
    {
      "id": "section-16",
      "title": "Continue exploring",
      "start": 7890,
      "end": 7913
    },
    {
      "id": "section-17",
      "title": "Foundations",
      "start": 7913,
      "end": 8324
    },
    {
      "id": "section-18",
      "title": "Related lessons",
      "start": 8324,
      "end": 8775
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#floating-point-reduction-order"
    },
    {
      "title": "NVIDIA CUDA 13.1.1 — Floating-Point Computation",
      "url": "https://docs.nvidia.com/cuda/archive/13.1.1/cuda-programming-guide/05-appendices/mathematical-functions.html"
    },
    {
      "title": "LLVM — Language Reference, fast-math rewrite flags",
      "url": "https://llvm.org/docs/LangRef.html#fast-math-flags"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/floating-point-reduction-order-b8b43418bbe78f69.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Multiply–accumulate and fused arithmetic",
      "url": "https://semiaifoundry.com/learn/atlas/mac/"
    },
    {
      "title": "Quantization and numerical representation",
      "url": "https://semiaifoundry.com/learn/atlas/quantization/"
    },
    {
      "title": "Compiler layout, lowering and buffer planning",
      "url": "https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/"
    },
    {
      "title": "Collective communication library",
      "url": "https://semiaifoundry.com/learn/atlas/nccl/"
    },
    {
      "title": "Sharded matrix multiplication and result ownership",
      "url": "https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/"
    },
    {
      "title": "Uneven batches and the global gradient",
      "url": "https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Systolic arrays and tensor dataflow",
      "url": "https://semiaifoundry.com/learn/atlas/systolic-array-dataflow/"
    }
  ],
  "recordHash": "86f6526b5e6b4f533a5f3be9c45c920549eadd2ac7ba48d689fc599d086b2d67",
  "contentFormat": "text/markdown",
  "content": "Executable software · Atlas v1.3.0\n\n# Floating-point reductions and compiler permissions\n\nDistinguish algebraic equivalence, numerical error and bitwise reproducibility when compilation or collectives regroup arithmetic.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#floating-point-reduction-order)Logical values and distributed ownership; no physical scale or vendor timing\n\nLearning objectiveReproduce both binary32 reduction orders and explain why compiler permissions, accuracy checks and reproducibility requirements must be specified separately.\n\n## The mechanism\n\nReal-number addition is associative, but a floating-point implementation rounds intermediate results. Changing the reduction tree changes where those rounding steps occur. A compiler that vectorizes a sum or a distributed library that groups rank contributions differently can therefore produce different low bits, and cancellation can make the difference much larger relative to the final result. LLVM exposes reassociation as an explicit fast-math permission; permission to contract a multiply and add is a separate choice. Such options should be read as part of the numerical contract, not assumed to be enabled in every program. A fixed reduction schedule can improve repeatability without establishing that its answer is the most accurate. Conversely, a numerically acceptable answer need not be bitwise identical to another legal result. The small integer schedules elsewhere in the atlas deliberately avoid this issue by keeping every operation exactly representable. This lesson changes that arithmetic assumption while keeping the underlying sum visible.\n\n1.  Specify input representation, accumulator dtype, rounding mode and treatment of fused operations before comparing outputs.\n2.  Write out the intermediate additions imposed by the chosen serial, vector or distributed reduction order.\n3.  Round after each stated operation rather than rounding only the final real-number sum.\n4.  Inspect compiler reassociation and contraction permissions, plus the library and rank topology that determine the actual reduction tree.\n5.  Compare with an independent higher-precision or exact reference where available; use an absolute tolerance near cancellation and a justified relative tolerance elsewhere.\n6.  Validate downstream loss or task behavior separately. A repeatability check and a numerical-accuracy check answer different questions.\n\n## Science and assumptions\n\n### Rounding makes grouping observable\n\n$$\n\\operatorname{fl}\\!\\left(\\operatorname{fl}(a+b)+c\\right)\\ne\\operatorname{fl}\\!\\left(a+\\operatorname{fl}(b+c)\\right)\\quad\\text{in general}\n$$\n\nRounding the inner addition before the outer one means regrouped additions can differ.\n\nSymbols and units\n\nfl\n\nOne binary32 round-to-nearest ties-to-even operation in this example\n\noperation\n\na, b, c\n\nExactly representable input operands\n\nvalue\n\nfl denotes one rounding to the specified floating-point format. Intermediate representability determines the result.\n\nAssumptions The example uses finite binary32 values with round-to-nearest, ties-to-even; no underflow, overflow, NaNs, stochastic rounding or fused multiply-add.\n\n### Spacing at the chosen magnitude\n\n$$\n\\Delta_{[2^{24},2^{25})}=2^{24-23}=2,\\qquad (2^{24}+1)-2^{24}=1\\quad\\text{over the reals}\n$$\n\nThe spacing above two to the twenty-fourth in binary32 is two. The exact real-number sum with one and its negative is one.\n\nSymbols and units\n\nΔ\n\nAdjacent representable spacing in the stated interval\n\nvalue\n\n24\n\nExponent at the chosen magnitude\n\ncount\n\n23\n\nStored binary32 fraction-bit count\n\ncount\n\nIn the binary32 interval \\[2²⁴,2²⁵), adjacent representable values are two apart. The halfway sum 2²⁴+1 rounds to the even lower significand.\n\nAssumptions ULP here means the spacing immediately above 2²⁴. Spacing immediately below that power of two is 1; it is not constant across the boundary.\n\n## Engineering example\n\nLet a=16,777,216=2²⁴, b=1 and c=−16,777,216, all representable in binary32. Group left: a+b is exactly 16,777,217 before rounding, halfway between representable 16,777,216 and 16,777,218. Ties-to-even gives 16,777,216; adding c then gives 0. Group right: b+c=−16,777,215 is representable, so adding a gives 1. The exact mathematical sum is 1. A deterministic left reduction repeats 0 and is reproducible, yet its absolute error is 1. A hypothetical acceptance condition |computed−reference|≤0.01 rejects 0 and accepts 1. That tolerance is an authored condition for this example, not an accuracy threshold for real training. Accumulating these three binary32 inputs with sufficiently precise binary64 additions produces 1; this does not guarantee that wider accumulation always eliminates error in larger problems.\n\n## Distinctions and tradeoffs\n\nBitwise repeatability does not prove closeness to the exact result, and algebraic equivalence does not prove bitwise identity.\n\nStricter ordering or wider accumulation can constrain transformations and consume resources. Their value depends on the actual error and reproducibility requirements; this example provides no throughput ranking.\n\nGradient reductions, attention sums and low-precision accumulations connect compiler decisions to optimization and model outputs. A faster schedule must still satisfy a declared numerical and task-quality contract.\n\n## Specifications and interfaces\n\nToy format\n\nIEEE-style binary32 rounding\n\n24 significand bits for normal values, nearest with ties to even.\n\nAlgebraic equality\n\nSame real-number expression\n\nDoes not guarantee identical rounded intermediates.\n\nCompiler permissions\n\nreassoc and contract differ\n\nAn enabled reassociation can change reduction grouping; contraction can change rounding boundaries.\n\n### Interfaces\n\nExpression and numerical contract Input\n\nOperands, dtype, rounding, grouping and optimization permissions\n\nChecked numerical result Output\n\nComputed value, reference difference and acceptance criterion\n\n## Check your understanding\n\n### With binary32 ties-to-even rounding, what is fl(fl(2²⁴+1)−2²⁴)?\n\n1.  0\n2.  1\n3.  2\n\nAnswer and reasoning\n\n0\n\nThe first addition rounds back to 2²⁴ because adjacent representable values above it are two apart; subtraction then cancels to zero.\n\n### Why does grouping 2²⁴+(1−2²⁴) instead give 1?\n\n1.  The inner result −16,777,215 is representable\n2.  The network repairs all numerical errors\n3.  The compiler changes the inputs\n\nAnswer and reasoning\n\nThe inner result −16,777,215 is representable\n\nBelow this power-of-two boundary the required integer is representable, so no information is lost in the inner sum.\n\n### Which evidence shows a result is accurate relative to a stated reference?\n\n1.  Only repeating the same bit pattern\n2.  An error check against the independent reference\n3.  Only using more ranks\n\nAnswer and reasoning\n\nAn error check against the independent reference\n\nRepeatability and rank count do not establish numerical accuracy.\n\n## Evidence and scope\n\nThe NVIDIA guide supports finite-precision nonassociativity, and LLVM specifies the cited optimization permissions. The exact operands, intermediate values and 0.01 tolerance are authored. Tests use binary32 conversions locally and do not constitute a GPU, compiler or multi-rank experiment.\n\nThe library contains related numerical models. This lesson’s explicit binary32 reduction counterexample is not an interactive floating-point emulator.\n\n## Sources\n\n-   [NVIDIA CUDA 13.1.1 — Floating-Point Computation](https://docs.nvidia.com/cuda/archive/13.1.1/cuda-programming-guide/05-appendices/mathematical-functions.html)\n-   [LLVM — Language Reference, fast-math rewrite flags](https://llvm.org/docs/LangRef.html#fast-math-flags)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/floating-point-reduction-order-b8b43418bbe78f69.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n-   [Quantization and numerical representation](https://semiaifoundry.com/learn/atlas/quantization/)\n-   [Compiler layout, lowering and buffer planning](https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/)\n-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)\n\n### Related lessons\n\n-   [Sharded matrix multiplication and result ownership](https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/)\n-   [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Systolic arrays and tensor dataflow](https://semiaifoundry.com/learn/atlas/systolic-array-dataflow/)"
}