{
  "schemaVersion": 1,
  "id": "atlas-lesson-optimizer-schedule-stability",
  "title": "Initialization, learning rate and stable updates",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-pretraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/optimizer-schedule-stability-6b89200eaa73f2ab.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#optimizer-schedule-stability",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "8da24fc06f95e4e9a91aa6bd4ab1829bc9e8d5bd56ff04caa4fc8e5d0a8563ec",
  "contentLength": 6234,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-optimizer-schedule-stability.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-optimizer-schedule-stability.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Initialization, learning rate and stable updates",
      "start": 44,
      "end": 453
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 453,
      "end": 1431
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1431,
      "end": 1459
    },
    {
      "id": "section-4",
      "title": "Initialization, learning rate and stable updates · worked relation",
      "start": 1459,
      "end": 2629
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2629,
      "end": 3045
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3045,
      "end": 3460
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3460,
      "end": 3921
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3921,
      "end": 4123
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4123,
      "end": 4152
    },
    {
      "id": "section-10",
      "title": "What preserves the direction of gradient \\[3,4\\] during norm clipping?",
      "start": 4152,
      "end": 4602
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4602,
      "end": 5279
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5279,
      "end": 5737
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5737,
      "end": 5760
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5760,
      "end": 5968
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5968,
      "end": 6234
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#optimizer-schedule-stability"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Decoupled Weight Decay Regularization",
      "url": "https://arxiv.org/html/1711.05101v3"
    },
    {
      "title": "PyTorch Adam algorithm",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/optimizer-schedule-stability-6b89200eaa73f2ab.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Residual stream & normalization",
      "url": "https://semiaifoundry.com/learn/atlas/residual-normalization/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Gradient accumulation under a fixed global objective",
      "url": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/"
    }
  ],
  "recordHash": "cf8d97bcce9c2d4a9694f3827b1bfa73045240a2d66ba3edf3aa7237642d59d5",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-pretraining · Atlas v1.3.0\n\n# Initialization, learning rate and stable updates\n\nSeparate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#optimizer-schedule-stability)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveSeparate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\n## The mechanism\n\nThe derivative specifies a local direction; the optimizer and schedule specify the actual state transition. Global norm clipping rescales an entire gradient vector before the update. Adam additionally carries first and second moments and corrects their initialization bias. AdamW applies a distinct shrinkage term rather than feeding an L2 derivative into those adaptive moments. A schedule changes the step size over updates, and its position belongs in a restart record. The lifecycle core executes clipping and constant-rate Adam or SGD; warmup, decay schedules and AdamW below are worked extensions, not hidden features of that core.\n\n1.  Compute the raw gradient norm before any clipping.\n2.  Apply one declared global scale to all gradient entries.\n3.  Update optimizer moments and their time counter from the clipped gradient.\n4.  Apply the stated learning rate and, only when selected by the algorithm, a separate decay term; compare fixed-seed runs.\n\n## Science and assumptions\n\n### Initialization, learning rate and stable updates · worked relation\n\n$$\ng_c=g\\min\\!\\left(1,\\frac{c}{\\lVert g\\rVert_2}\\right),\\qquad \\theta^+=\\theta-\\eta u-\\eta\\lambda\\theta\n$$\n\nSeparate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\nSymbols and units\n\ng,g\\_c\n\nRaw and globally clipped gradient vectors\n\nloss per parameter\n\nc\n\nGradient norm limit\n\nloss per parameter\n\ntheta\n\nParameter vector\n\ndimensionless\n\neta\n\nStep size in the declared numerical convention\n\ndimensionless\n\nu\n\nAdaptive update direction\n\ndimensionless\n\nlambda\n\nDecoupled decay coefficient\n\ndimensionless\n\nFor raw gradient \\[3,4\\], norm is 5. A clip threshold of 1 gives \\[0.6,0.8\\]. With initial weights \\[2,2\\] and SGD rate 0.1, the result is \\[1.94,1.92\\]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.\n\nAssumptions For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.\n\n## Engineering example\n\nFor raw gradient \\[3,4\\], norm is 5. A clip threshold of 1 gives \\[0.6,0.8\\]. With initial weights \\[2,2\\] and SGD rate 0.1, the result is \\[1.94,1.92\\]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Clamping each entry independently to 1.\n\nClipping can contain extreme updates but can also suppress useful signal. Stability on a small run does not determine a universally good schedule.\n\nTurn an objective into a stable, restartable experiment with controlled budgets. Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable clipping; worked schedule and AdamW\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nSeparate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\nFor zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.\n\n### Interfaces\n\nInputs and state Input\n\nCompute the raw gradient norm before any clipping.\n\nResult and check Output\n\nSeparate gradient clipping, adaptive moments, decay and the learning-rate schedule.\n\n## Check your understanding\n\n### What preserves the direction of gradient \\[3,4\\] during norm clipping?\n\n1.  Clamping each entry independently to 1.\n2.  Replacing the gradient with \\[1,0\\].\n3.  Multiplying both entries by 1/5.\n\nAnswer and reasoning\n\nMultiplying both entries by 1/5.\n\nMultiplying both entries by 1/5. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.\n\n## Evidence and scope\n\nMode: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.\n\n## Sources\n\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n-   [Decoupled Weight Decay Regularization](https://arxiv.org/html/1711.05101v3)\n-   [PyTorch Adam algorithm](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/optimizer-schedule-stability-6b89200eaa73f2ab.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)\n\n### Related lessons\n\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)"
}