# Floating-point reductions and compiler permissions Canonical: https://semiaifoundry.com/learn/atlas/floating-point-reduction-order/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Executable software · Atlas v1.3.0 # Floating-point reductions and compiler permissions Distinguish algebraic equivalence, numerical error and bitwise reproducibility when compilation or collectives regroup arithmetic. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#floating-point-reduction-order)Logical values and distributed ownership; no physical scale or vendor timing Learning objectiveReproduce both binary32 reduction orders and explain why compiler permissions, accuracy checks and reproducibility requirements must be specified separately. ## The mechanism Real-number addition is associative, but a floating-point implementation rounds intermediate results. Changing the reduction tree changes where those rounding steps occur. A compiler that vectorizes a sum or a distributed library that groups rank contributions differently can therefore produce different low bits, and cancellation can make the difference much larger relative to the final result. LLVM exposes reassociation as an explicit fast-math permission; permission to contract a multiply and add is a separate choice. Such options should be read as part of the numerical contract, not assumed to be enabled in every program. A fixed reduction schedule can improve repeatability without establishing that its answer is the most accurate. Conversely, a numerically acceptable answer need not be bitwise identical to another legal result. The small integer schedules elsewhere in the atlas deliberately avoid this issue by keeping every operation exactly representable. This lesson changes that arithmetic assumption while keeping the underlying sum visible. 1. Specify input representation, accumulator dtype, rounding mode and treatment of fused operations before comparing outputs. 2. Write out the intermediate additions imposed by the chosen serial, vector or distributed reduction order. 3. Round after each stated operation rather than rounding only the final real-number sum. 4. Inspect compiler reassociation and contraction permissions, plus the library and rank topology that determine the actual reduction tree. 5. Compare with an independent higher-precision or exact reference where available; use an absolute tolerance near cancellation and a justified relative tolerance elsewhere. 6. Validate downstream loss or task behavior separately. A repeatability check and a numerical-accuracy check answer different questions. ## Science and assumptions ### Rounding makes grouping observable $$ \operatorname{fl}\!\left(\operatorname{fl}(a+b)+c\right)\ne\operatorname{fl}\!\left(a+\operatorname{fl}(b+c)\right)\quad\text{in general} $$ Rounding the inner addition before the outer one means regrouped additions can differ. Symbols and units fl One binary32 round-to-nearest ties-to-even operation in this example operation a, b, c Exactly representable input operands value fl denotes one rounding to the specified floating-point format. Intermediate representability determines the result. Assumptions The example uses finite binary32 values with round-to-nearest, ties-to-even; no underflow, overflow, NaNs, stochastic rounding or fused multiply-add. ### Spacing at the chosen magnitude $$ \Delta_{[2^{24},2^{25})}=2^{24-23}=2,\qquad (2^{24}+1)-2^{24}=1\quad\text{over the reals} $$ The spacing above two to the twenty-fourth in binary32 is two. The exact real-number sum with one and its negative is one. Symbols and units Δ Adjacent representable spacing in the stated interval value 24 Exponent at the chosen magnitude count 23 Stored binary32 fraction-bit count count In the binary32 interval \[2²⁴,2²⁵), adjacent representable values are two apart. The halfway sum 2²⁴+1 rounds to the even lower significand. Assumptions ULP here means the spacing immediately above 2²⁴. Spacing immediately below that power of two is 1; it is not constant across the boundary. ## Engineering example Let a=16,777,216=2²⁴, b=1 and c=−16,777,216, all representable in binary32. Group left: a+b is exactly 16,777,217 before rounding, halfway between representable 16,777,216 and 16,777,218. Ties-to-even gives 16,777,216; adding c then gives 0. Group right: b+c=−16,777,215 is representable, so adding a gives 1. The exact mathematical sum is 1. A deterministic left reduction repeats 0 and is reproducible, yet its absolute error is 1. A hypothetical acceptance condition |computed−reference|≤0.01 rejects 0 and accepts 1. That tolerance is an authored condition for this example, not an accuracy threshold for real training. Accumulating these three binary32 inputs with sufficiently precise binary64 additions produces 1; this does not guarantee that wider accumulation always eliminates error in larger problems. ## Distinctions and tradeoffs Bitwise repeatability does not prove closeness to the exact result, and algebraic equivalence does not prove bitwise identity. Stricter ordering or wider accumulation can constrain transformations and consume resources. Their value depends on the actual error and reproducibility requirements; this example provides no throughput ranking. Gradient reductions, attention sums and low-precision accumulations connect compiler decisions to optimization and model outputs. A faster schedule must still satisfy a declared numerical and task-quality contract. ## Specifications and interfaces Toy format IEEE-style binary32 rounding 24 significand bits for normal values, nearest with ties to even. Algebraic equality Same real-number expression Does not guarantee identical rounded intermediates. Compiler permissions reassoc and contract differ An enabled reassociation can change reduction grouping; contraction can change rounding boundaries. ### Interfaces Expression and numerical contract Input Operands, dtype, rounding, grouping and optimization permissions Checked numerical result Output Computed value, reference difference and acceptance criterion ## Check your understanding ### With binary32 ties-to-even rounding, what is fl(fl(2²⁴+1)−2²⁴)? 1. 0 2. 1 3. 2 Answer and reasoning 0 The first addition rounds back to 2²⁴ because adjacent representable values above it are two apart; subtraction then cancels to zero. ### Why does grouping 2²⁴+(1−2²⁴) instead give 1? 1. The inner result −16,777,215 is representable 2. The network repairs all numerical errors 3. The compiler changes the inputs Answer and reasoning The inner result −16,777,215 is representable Below this power-of-two boundary the required integer is representable, so no information is lost in the inner sum. ### Which evidence shows a result is accurate relative to a stated reference? 1. Only repeating the same bit pattern 2. An error check against the independent reference 3. Only using more ranks Answer and reasoning An error check against the independent reference Repeatability and rank count do not establish numerical accuracy. ## Evidence and scope The NVIDIA guide supports finite-precision nonassociativity, and LLVM specifies the cited optimization permissions. The exact operands, intermediate values and 0.01 tolerance are authored. Tests use binary32 conversions locally and do not constitute a GPU, compiler or multi-rank experiment. The library contains related numerical models. This lesson’s explicit binary32 reduction counterexample is not an interactive floating-point emulator. ## Sources - [NVIDIA CUDA 13.1.1 — Floating-Point Computation](https://docs.nvidia.com/cuda/archive/13.1.1/cuda-programming-guide/05-appendices/mathematical-functions.html) - [LLVM — Language Reference, fast-math rewrite flags](https://llvm.org/docs/LangRef.html#fast-math-flags) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/floating-point-reduction-order-b8b43418bbe78f69.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/) - [Quantization and numerical representation](https://semiaifoundry.com/learn/atlas/quantization/) - [Compiler layout, lowering and buffer planning](https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/) - [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/) ### Related lessons - [Sharded matrix multiplication and result ownership](https://semiaifoundry.com/learn/atlas/sharded-matmul-ownership/) - [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) - [Systolic arrays and tensor dataflow](https://semiaifoundry.com/learn/atlas/systolic-array-dataflow/)