The mechanism
Gradient accumulation is a way to evaluate one global objective through several smaller forward/backward passes. Equivalence requires the same parameters for all those passes and the same weighting of included targets. Averaging two microbatch means equally silently changes the objective when their target counts differ. Optimizer steps between microbatches change the parameters at which later derivatives are evaluated. Nonlinear operations such as norm clipping also fail to commute with accumulation. Randomness, stateful layers and rounding add further limits to exact equivalence.
- Freeze the current weights across all microbatches in the update.
- Retain each microbatch loss sum and valid-target count.
- Add gradient sums and divide once by the global target denominator.
- Clip the resulting global gradient once, then make exactly one optimizer step.
Science and assumptions
Gradient accumulation under a fixed global objective · worked relation
Recover a full-batch gradient from unequal target counts before clipping or stepping.
Symbols and units
- g_k
- Mean gradient for microbatch k
loss per parameter
- n_k
- Valid targets in microbatch k
tokens
- g
- Global token-mean gradient
loss per parameter
- theta
- Parameter evaluated consistently across microbatches
dimensionless
- eta
- SGD step size
dimensionless
Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.
Assumptions All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.
Engineering example
Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.
Distinctions and tradeoffs
It is a mistake to conclude: Once per microbatch before summation.
Microbatches lower activation demand at the cost of extra scheduling and stricter state bookkeeping. They do not automatically preserve the objective.
Turn an objective into a stable, restartable experiment with controlled budgets. Recover a full-batch gradient from unequal target counts before clipping or stepping.
Specifications and interfaces
- Learning mode
- worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Recover a full-batch gradient from unequal target counts before clipping or stepping.
All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.
Interfaces
- Inputs and state Input
- Freeze the current weights across all microbatches in the update.
- Result and check Output
- Recover a full-batch gradient from unequal target counts before clipping or stepping.
Check your understanding
When should clipping occur to reproduce the declared full-batch update?
- After target-weighted accumulation and before the single optimizer step.
- Once per microbatch before summation.
- After two separate optimizer steps.
Answer and reasoning
After target-weighted accumulation and before the single optimizer step.
After target-weighted accumulation and before the single optimizer step. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.
Evidence and scope
Mode: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.