{
  "schemaVersion": 1,
  "id": "atlas-lesson-microbatch-equivalent-updates",
  "title": "Gradient accumulation under a fixed global objective",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Recover a full-batch gradient from unequal target counts before clipping or stepping.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-pretraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/microbatch-equivalent-updates-e06cc42040fac364.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#microbatch-equivalent-updates",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "a5b74f40fe3e160d9bb9daf68218515e4244196b4cb26081eca39da4200f2803",
  "contentLength": 6127,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-microbatch-equivalent-updates.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-microbatch-equivalent-updates.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Gradient accumulation under a fixed global objective",
      "start": 44,
      "end": 462
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 462,
      "end": 1355
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1355,
      "end": 1383
    },
    {
      "id": "section-4",
      "title": "Gradient accumulation under a fixed global objective · worked relation",
      "start": 1383,
      "end": 2480
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2480,
      "end": 2867
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2867,
      "end": 3286
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3286,
      "end": 3737
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3737,
      "end": 3956
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3956,
      "end": 3985
    },
    {
      "id": "section-10",
      "title": "When should clipping occur to reproduce the declared full-batch update?",
      "start": 3985,
      "end": 4573
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4573,
      "end": 5226
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5226,
      "end": 5620
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5620,
      "end": 5643
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5643,
      "end": 5866
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5866,
      "end": 6127
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#microbatch-equivalent-updates"
    },
    {
      "title": "TRL SFT Trainer",
      "url": "https://huggingface.co/docs/trl/sft_trainer"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/microbatch-equivalent-updates-e06cc42040fac364.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Uneven batches and the global gradient",
      "url": "https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Initialization, learning rate and stable updates",
      "url": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/"
    }
  ],
  "recordHash": "f60c05d4f4bb3227ba1405d4cbfbb6e90206338b750b504f8af7b9d7d1e56874",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-pretraining · Atlas v1.3.0\n\n# Gradient accumulation under a fixed global objective\n\nRecover a full-batch gradient from unequal target counts before clipping or stepping.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#microbatch-equivalent-updates)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveRecover a full-batch gradient from unequal target counts before clipping or stepping.\n\n## The mechanism\n\nGradient accumulation is a way to evaluate one global objective through several smaller forward/backward passes. Equivalence requires the same parameters for all those passes and the same weighting of included targets. Averaging two microbatch means equally silently changes the objective when their target counts differ. Optimizer steps between microbatches change the parameters at which later derivatives are evaluated. Nonlinear operations such as norm clipping also fail to commute with accumulation. Randomness, stateful layers and rounding add further limits to exact equivalence.\n\n1.  Freeze the current weights across all microbatches in the update.\n2.  Retain each microbatch loss sum and valid-target count.\n3.  Add gradient sums and divide once by the global target denominator.\n4.  Clip the resulting global gradient once, then make exactly one optimizer step.\n\n## Science and assumptions\n\n### Gradient accumulation under a fixed global objective · worked relation\n\n$$\ng=\\frac{\\sum_k n_k g_k}{\\sum_k n_k},\\qquad \\theta^+=\\theta-\\eta\\operatorname{clip}(g)\n$$\n\nRecover a full-batch gradient from unequal target counts before clipping or stepping.\n\nSymbols and units\n\ng\\_k\n\nMean gradient for microbatch k\n\nloss per parameter\n\nn\\_k\n\nValid targets in microbatch k\n\ntokens\n\ng\n\nGlobal token-mean gradient\n\nloss per parameter\n\ntheta\n\nParameter evaluated consistently across microbatches\n\ndimensionless\n\neta\n\nSGD step size\n\ndimensionless\n\nTwo valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.\n\nAssumptions All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.\n\n## Engineering example\n\nTwo valid targets yield mean scalar gradient 3; five yield mean gradient −1. The global mean is (2×3+5×(−1))/7=1/7. Equally averaging microbatch means gives 1 and is wrong. If each local mean is clipped to magnitude 1 first, the weighted result becomes (2−5)/7=−3/7, even reversing the sign. At weight 2 and SGD rate 0.7, the correct single update produces 1.9.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Once per microbatch before summation.\n\nMicrobatches lower activation demand at the cost of extra scheduling and stricter state bookkeeping. They do not automatically preserve the objective.\n\nTurn an objective into a stable, restartable experiment with controlled budgets. Recover a full-batch gradient from unequal target counts before clipping or stepping.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nRecover a full-batch gradient from unequal target counts before clipping or stepping.\n\nAll gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.\n\n### Interfaces\n\nInputs and state Input\n\nFreeze the current weights across all microbatches in the update.\n\nResult and check Output\n\nRecover a full-batch gradient from unequal target counts before clipping or stepping.\n\n## Check your understanding\n\n### When should clipping occur to reproduce the declared full-batch update?\n\n1.  After target-weighted accumulation and before the single optimizer step.\n2.  Once per microbatch before summation.\n3.  After two separate optimizer steps.\n\nAnswer and reasoning\n\nAfter target-weighted accumulation and before the single optimizer step.\n\nAfter target-weighted accumulation and before the single optimizer step. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.\n\n## Evidence and scope\n\nMode: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. All gradients evaluated at the same weights with the same stochastic choices needed for equivalence. Exact arithmetic example; finite reductions can introduce rounding differences.\n\n## Sources\n\n-   [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/microbatch-equivalent-updates-e06cc42040fac364.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n\n### Related lessons\n\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)"
}