{"id":"microbatch-equivalent-updates","name":"Gradient accumulation under a fixed global objective","shortName":"Gradient accumulation under a fixed global objective","role":"Recover a full-batch gradient from unequal target counts before clipping or stepping.","description":"Gradient accumulation is a way to evaluate one global objective through several smaller forward/backward passes. Equivalence requires the same parameters for all those passes and the same weighting of included targets. Averaging two microbatch means equally silently changes the objective when their target counts differ. Optimizer steps between microbatches change the parameters at which later derivatives are evaluated. Nonlinear operations such as norm clipping also fail to commute with accumulation. Randomness, stateful layers and rounding add further limits to exact equivalence.","mechanism":["Freeze the current weights across all microbatches in the update.","Retain each microbatch loss sum and valid-target count.","Add gradient sums and divide once by the global target denominator.","Clip the resulting global gradient once, then make exactly one optimizer step."],"learningObjective":"Recover a full-batch gradient from unequal target counts before clipping or stepping.","engineeringExample":"Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.","evidenceNotes":"Mode: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Recover a full-batch gradient from unequal target counts before clipping or stepping.","context":"All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Freeze the current weights across all microbatches in the update."},{"name":"Result and check","direction":"out","signal":"Recover a full-batch gradient from unequal target counts before clipping or stepping."}],"tradeoff":"Microbatches lower activation demand at the cost of extra scheduling and stricter state bookkeeping. They do not automatically preserve the objective.","whyAI":"Turn an objective into a stable, restartable experiment with controlled budgets. Recover a full-batch gradient from unequal target counts before clipping or stepping.","misconception":"It is a mistake to conclude: Once per microbatch before summation.","check":{"question":"When should clipping occur to reproduce the declared full-batch update?","options":["After target-weighted accumulation and before the single optimizer step.","Once per microbatch before summation.","After two separate optimizer steps."],"answer":0,"explanation":"After target-weighted accumulation and before the single optimizer step. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences."},"prerequisites":["distributed-gradient-weighting","loss-optimizer"],"related":["distributed-gradient-weighting","loss-optimizer","causal-pretraining-objective","optimizer-schedule-stability"],"sources":[{"title":"TRL SFT Trainer","url":"https://huggingface.co/docs/trl/sft_trainer"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-pretraining","subbranch":"Gradient accumulation under a fixed global objective","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.","science":[{"title":"Gradient accumulation under a fixed global objective · worked relation","equation":"g=\\frac{\\sum_k n_k g_k}{\\sum_k n_k},\\qquad \\theta^+=\\theta-\\eta\\operatorname{clip}(g)","explanation":"Two valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.","assumptions":"All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences."}],"history":[]}