# Gradient accumulation under a fixed global objective Canonical: https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-pretraining · Atlas v1.3.0 # Gradient accumulation under a fixed global objective Recover a full-batch gradient from unequal target counts before clipping or stepping. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#microbatch-equivalent-updates)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveRecover a full-batch gradient from unequal target counts before clipping or stepping. ## The mechanism Gradient accumulation is a way to evaluate one global objective through several smaller forward/backward passes. Equivalence requires the same parameters for all those passes and the same weighting of included targets. Averaging two microbatch means equally silently changes the objective when their target counts differ. Optimizer steps between microbatches change the parameters at which later derivatives are evaluated. Nonlinear operations such as norm clipping also fail to commute with accumulation. Randomness, stateful layers and rounding add further limits to exact equivalence. 1. Freeze the current weights across all microbatches in the update. 2. Retain each microbatch loss sum and valid-target count. 3. Add gradient sums and divide once by the global target denominator. 4. Clip the resulting global gradient once, then make exactly one optimizer step. ## Science and assumptions ### Gradient accumulation under a fixed global objective · worked relation $$ g=\frac{\sum_k n_k g_k}{\sum_k n_k},\qquad \theta^+=\theta-\eta\operatorname{clip}(g) $$ Recover a full-batch gradient from unequal target counts before clipping or stepping. Symbols and units g\_k Mean gradient for microbatch k loss per parameter n\_k Valid targets in microbatch k tokens g Global token-mean gradient loss per parameter theta Parameter evaluated consistently across microbatches dimensionless eta SGD step size dimensionless Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9. Assumptions All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. ## Engineering example Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9. ## Distinctions and tradeoffs It is a mistake to conclude: Once per microbatch before summation. Microbatches lower activation demand at the cost of extra scheduling and stricter state bookkeeping. They do not automatically preserve the objective. Turn an objective into a stable, restartable experiment with controlled budgets. Recover a full-batch gradient from unequal target counts before clipping or stepping. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Recover a full-batch gradient from unequal target counts before clipping or stepping. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. ### Interfaces Inputs and state Input Freeze the current weights across all microbatches in the update. Result and check Output Recover a full-batch gradient from unequal target counts before clipping or stepping. ## Check your understanding ### When should clipping occur to reproduce the declared full-batch update? 1. After target-weighted accumulation and before the single optimizer step. 2. Once per microbatch before summation. 3. After two separate optimizer steps. Answer and reasoning After target-weighted accumulation and before the single optimizer step. After target-weighted accumulation and before the single optimizer step. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. ## Evidence and scope Mode: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. ## Sources - [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer) - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/microbatch-equivalent-updates-e06cc42040fac364.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/) - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) ### Related lessons - [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/) - [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)