The mechanism
Real-number addition is associative, but a floating-point implementation rounds intermediate results. Changing the reduction tree changes where those rounding steps occur. A compiler that vectorizes a sum or a distributed library that groups rank contributions differently can therefore produce different low bits, and cancellation can make the difference much larger relative to the final result. LLVM exposes reassociation as an explicit fast-math permission; permission to contract a multiply and add is a separate choice. Such options should be read as part of the numerical contract, not assumed to be enabled in every program. A fixed reduction schedule can improve repeatability without establishing that its answer is the most accurate. Conversely, a numerically acceptable answer need not be bitwise identical to another legal result. The small integer schedules elsewhere in the atlas deliberately avoid this issue by keeping every operation exactly representable. This lesson changes that arithmetic assumption while keeping the underlying sum visible.
- Specify input representation, accumulator dtype, rounding mode and treatment of fused operations before comparing outputs.
- Write out the intermediate additions imposed by the chosen serial, vector or distributed reduction order.
- Round after each stated operation rather than rounding only the final real-number sum.
- Inspect compiler reassociation and contraction permissions, plus the library and rank topology that determine the actual reduction tree.
- Compare with an independent higher-precision or exact reference where available; use an absolute tolerance near cancellation and a justified relative tolerance elsewhere.
- Validate downstream loss or task behavior separately. A repeatability check and a numerical-accuracy check answer different questions.
Science and assumptions
Rounding makes grouping observable
Rounding the inner addition before the outer one means regrouped additions can differ.
Symbols and units
- fl
- One binary32 round-to-nearest ties-to-even operation in this example
operation
- a, b, c
- Exactly representable input operands
value
fl denotes one rounding to the specified floating-point format. Intermediate representability determines the result.
Assumptions The example uses finite binary32 values with round-to-nearest, ties-to-even; no underflow, overflow, NaNs, stochastic rounding or fused multiply-add.
Spacing at the chosen magnitude
The spacing above two to the twenty-fourth in binary32 is two. The exact real-number sum with one and its negative is one.
Symbols and units
- Δ
- Adjacent representable spacing in the stated interval
value
- 24
- Exponent at the chosen magnitude
count
- 23
- Stored binary32 fraction-bit count
count
In the binary32 interval [2²⁴,2²⁵), adjacent representable values are two apart. The halfway sum 2²⁴+1 rounds to the even lower significand.
Assumptions ULP here means the spacing immediately above 2²⁴. Spacing immediately below that power of two is 1; it is not constant across the boundary.
Engineering example
Let a=16,777,216=2²⁴, b=1 and c=−16,777,216, all representable in binary32. Group left: a+b is exactly 16,777,217 before rounding, halfway between representable 16,777,216 and 16,777,218. Ties-to-even gives 16,777,216; adding c then gives 0. Group right: b+c=−16,777,215 is representable, so adding a gives 1. The exact mathematical sum is 1. A deterministic left reduction repeats 0 and is reproducible, yet its absolute error is 1. A hypothetical acceptance condition |computed−reference|≤0.01 rejects 0 and accepts 1. That tolerance is an authored condition for this example, not an accuracy threshold for real training. Accumulating these three binary32 inputs with sufficiently precise binary64 additions produces 1; this does not guarantee that wider accumulation always eliminates error in larger problems.
Distinctions and tradeoffs
Bitwise repeatability does not prove closeness to the exact result, and algebraic equivalence does not prove bitwise identity.
Stricter ordering or wider accumulation can constrain transformations and consume resources. Their value depends on the actual error and reproducibility requirements; this example provides no throughput ranking.
Gradient reductions, attention sums and low-precision accumulations connect compiler decisions to optimization and model outputs. A faster schedule must still satisfy a declared numerical and task-quality contract.
Specifications and interfaces
- Toy format
- IEEE-style binary32 rounding
24 significand bits for normal values, nearest with ties to even.
- Algebraic equality
- Same real-number expression
Does not guarantee identical rounded intermediates.
- Compiler permissions
- reassoc and contract differ
An enabled reassociation can change reduction grouping; contraction can change rounding boundaries.
Interfaces
- Expression and numerical contract Input
- Operands, dtype, rounding, grouping and optimization permissions
- Checked numerical result Output
- Computed value, reference difference and acceptance criterion
Check your understanding
With binary32 ties-to-even rounding, what is fl(fl(2²⁴+1)−2²⁴)?
- 0
- 1
- 2
Answer and reasoning
0
The first addition rounds back to 2²⁴ because adjacent representable values above it are two apart; subtraction then cancels to zero.
Why does grouping 2²⁴+(1−2²⁴) instead give 1?
- The inner result −16,777,215 is representable
- The network repairs all numerical errors
- The compiler changes the inputs
Answer and reasoning
The inner result −16,777,215 is representable
Below this power-of-two boundary the required integer is representable, so no information is lost in the inner sum.
Which evidence shows a result is accurate relative to a stated reference?
- Only repeating the same bit pattern
- An error check against the independent reference
- Only using more ranks
Answer and reasoning
An error check against the independent reference
Repeatability and rank count do not establish numerical accuracy.
Evidence and scope
The NVIDIA guide supports finite-precision nonassociativity, and LLVM specifies the cited optimization permissions. The exact operands, intermediate values and 0.01 tolerance are authored. Tests use binary32 conversions locally and do not constitute a GPU, compiler or multi-rank experiment.
The library contains related numerical models. This lesson’s explicit binary32 reduction counterexample is not an interactive floating-point emulator.
Sources
Continue exploring
Foundations
- Multiply–accumulate and fused arithmetic
- Quantization and numerical representation
- Compiler layout, lowering and buffer planning
- Collective communication library