{
  "schemaVersion": 1,
  "id": "atlas-lesson-loss-optimizer",
  "title": "Loss, gradients & optimizer",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Turn prediction errors into parameter updates.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/loss-optimizer/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Learning",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This follows the ordinary Adam equations in the linked paper. The gradient is supplied rather than derived here; clipping, weight decay, loss scaling and sharded optimizer state are excluded. Computing a gradient and applying an update remain separate steps.\n\nThe threshold arithmetic is authored. The existing sensor experience executes actual logistic updates, while these three probabilities are a separate worked decision example.\n\nMode: executable derived-gradient optimizer. No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/loss-optimizer-75b6c32e6a4f283f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#loss-optimizer",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ddbc8617107da8278f984b282ddc898917f573899b7525d02b8846ec4009dc8f",
  "contentLength": 11602,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-loss-optimizer.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-loss-optimizer.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Loss, gradients & optimizer",
      "start": 25,
      "end": 327
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 327,
      "end": 2344
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2344,
      "end": 2372
    },
    {
      "id": "section-4",
      "title": "Adam adapts updates using gradient moments",
      "start": 2372,
      "end": 4456
    },
    {
      "id": "section-5",
      "title": "Compute gradients before applying the optimizer · worked relation",
      "start": 4456,
      "end": 5643
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 5643,
      "end": 6671
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 6671,
      "end": 7006
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 7006,
      "end": 7185
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 7185,
      "end": 7305
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 7305,
      "end": 7334
    },
    {
      "id": "section-11",
      "title": "Which process changes learned weights in a normal training step?",
      "start": 7334,
      "end": 7610
    },
    {
      "id": "section-12",
      "title": "A decision threshold changes from 0.5 to 0.7 with fixed model outputs. What necessarily changes?",
      "start": 7610,
      "end": 8006
    },
    {
      "id": "section-13",
      "title": "Why is the first Adam change about 0.1 rather than 0.05?",
      "start": 8006,
      "end": 8529
    },
    {
      "id": "section-14",
      "title": "Historical context",
      "start": 8529,
      "end": 8552
    },
    {
      "id": "section-15",
      "title": "2014 · Adam combines momentum with adaptive scaling",
      "start": 8552,
      "end": 8877
    },
    {
      "id": "section-16",
      "title": "Evidence and scope",
      "start": 8877,
      "end": 9854
    },
    {
      "id": "section-17",
      "title": "Sources",
      "start": 9854,
      "end": 10506
    },
    {
      "id": "section-18",
      "title": "Continue exploring",
      "start": 10506,
      "end": 10529
    },
    {
      "id": "section-19",
      "title": "Foundations",
      "start": 10529,
      "end": 10711
    },
    {
      "id": "section-20",
      "title": "Related lessons",
      "start": 10711,
      "end": 11602
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#loss-optimizer"
    },
    {
      "title": "Kingma & Ba — Adam: A Method for Stochastic Optimization",
      "url": "https://arxiv.org/abs/1412.6980"
    },
    {
      "title": "PyTorch — CrossEntropyLoss",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html"
    },
    {
      "title": "scikit-learn · Common pitfalls and data leakage",
      "url": "https://scikit-learn.org/stable/common_pitfalls.html#data-leakage"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "PyTorch Adam algorithm",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/loss-optimizer-75b6c32e6a4f283f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Residual stream & normalization",
      "url": "https://semiaifoundry.com/learn/atlas/residual-normalization/"
    },
    {
      "title": "Data, tensor, pipeline, and state parallelism",
      "url": "https://semiaifoundry.com/learn/atlas/parallelism/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Model artifacts, rollout and drift",
      "url": "https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Initialization, learning rate and stable updates",
      "url": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/"
    },
    {
      "title": "Gradient accumulation under a fixed global objective",
      "url": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/"
    }
  ],
  "recordHash": "2b505d4c1b11fd12ba6a7300e0fdebbef2e16af440ab5e3058c782e6cf3e419b",
  "contentFormat": "text/markdown",
  "content": "Learning · Atlas v1.3.0\n\n# Loss, gradients & optimizer\n\nTurn prediction errors into parameter updates.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#loss-optimizer)Logical structure · mapped onto physical hardware\n\nLearning objectiveTrace one Adam update from a gradient through moment state to a changed parameter.\n\n## The mechanism\n\nA loss assigns a numerical training objective to the model’s predictions. Backpropagation applies the chain rule through the computation graph to obtain gradients. An optimizer uses those gradients and, often, additional state to update weights. Adam maintains estimates related to gradient moments. This backward computation and optimizer storage distinguish training from ordinary inference. Distributed execution must coordinate the appropriate gradients or shards before consistent updates can proceed.\n\nA training objective, a decision threshold and an operational acceptance criterion answer different questions. Updating weights changes the predictive function; moving a threshold changes decisions for the same probabilities. Neither operation should be concealed inside an evaluation comparison. A frozen artifact and separate test set make these changes traceable into later deployment decisions.\n\nAn optimizer consumes a derived gradient; a supplied-gradient calculator teaches only the last part of that chain. The lifecycle experience adds an actual decoder loss and reverse pass before applying either SGD or Adam. Adam’s moments and step counter are learning state, not display history. Global norm clipping occurs before those moments. Changing the objective changes the gradient even if the optimizer setting is held fixed.\n\n1.  Run the forward computation and evaluate the loss.\n2.  Backpropagate derivatives through the recorded operations.\n3.  Combine required distributed information and update parameters.\n4.  The optimizer updates model parameters using the defined training loss and regularization.\n5.  Evaluation keeps thresholds and accepted error tradeoffs explicit instead of claiming that lower loss necessarily produces a better application.\n6.  Compute the gradient from the current batch loss and retain its denominator.\n7.  Apply the declared global clipping rule.\n8.  Update moments, bias-correct them and apply the chosen learning rate; save all resulting state.\n\n## Science and assumptions\n\n### Adam adapts updates using gradient moments\n\n$$\n\\begin{aligned}g_t&=\\nabla_\\theta\\mathcal L_t,\\\\m_t&=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\\\v_t&=\\beta_2v_{t-1}+(1-\\beta_2)g_t^{\\odot2},\\\\\\hat m_t&=\\frac{m_t}{1-\\beta_1^t},\\qquad\\hat v_t=\\frac{v_t}{1-\\beta_2^t},\\\\\\theta_{t+1}&=\\theta_t-\\eta\\frac{\\hat m_t}{\\sqrt{\\hat v_t}+\\varepsilon}\\end{aligned}\n$$\n\nAdam smooths gradients and their elementwise squares, corrects zero-initialization bias, then subtracts the step size times corrected first moment divided elementwise by square root of corrected second moment plus a stabilizer.\n\nAll squares, roots and divisions are elementwise. The two bias-correction relations are made explicit from the original assumptions. Parameter labels retain the source convention of updating θ\\_t to θ\\_(t+1).\n\nSymbols and units\n\nθₜ\n\nParameter vector before update t\n\nparameter-coordinate units\n\nℒₜ\n\nStep-t loss evaluated at the current parameters\n\nloss units\n\ngₜ\n\nGradient of step loss with respect to parameters\n\nloss units per parameter-coordinate unit\n\nmₜ,m̂ₜ\n\nFirst moment and its zero-initialization bias correction\n\nsame units as gₜ\n\nvₜ,v̂ₜ\n\nSecond raw moment and its bias correction\n\nsquared gradient units\n\nβ₁,β₂\n\nMoment decay coefficients\n\nfractions less than one\n\nη\n\nStep size applied to the normalized direction\n\nparameter-coordinate units\n\nε\n\nPositive denominator stabilizer\n\nsame units as square root of v̂ₜ\n\nt\n\nPositive update index for moment bias correction\n\ninteger index\n\n⊙2\n\nElementwise square\n\noperator\n\nThe first moment m smooths gradient direction; the second moment v tracks squared gradient magnitude. Bias-corrected estimates m̂ and v̂ scale each coordinate’s step. The optimizer consequently needs state arrays in addition to weights and gradients, connecting a mathematical update rule directly to accelerator memory requirements.\n\nAssumptions Products, squares, and divisions are coordinate-wise. With zero initialization, m̂ₜ=mₜ/(1−β₁ᵗ) and v̂ₜ=vₜ/(1−β₂ᵗ). This is ordinary Adam; decoupled weight decay, clipping, loss scaling, and distributed sharding require additional rules.\n\n### Compute gradients before applying the optimizer · worked relation\n\n$$\nm_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\quad v_t=\\beta_2v_{t-1}+(1-\\beta_2)g_t^2,\\quad\\theta^+=\\theta-\\eta\\frac{m_t/(1-\\beta_1^t)}{\\sqrt{v_t/(1-\\beta_2^t)}+\\epsilon}\n$$\n\nVerify the first actual gradient-driven SGD and Adam updates under explicit state.\n\nSymbols and units\n\nm\\_t\n\nFirst gradient moment\n\ngradient units\n\nv\\_t\n\nSecond gradient moment\n\nsquared gradient units\n\ng\\_t\n\nGradient after declared clipping\n\ngradient units\n\nbeta\\_1,beta\\_2\n\nMoment decay factors\n\ndimensionless\n\nt\n\nOptimizer update count\n\nupdates\n\neta\n\nLearning rate\n\ndimensionless\n\nepsilon\n\nDenominator stabilizer\n\ngradient units\n\ntheta\n\nParameter\n\ndimensionless\n\nFor a scalar gradient g=0.5, weight 2 and learning rate 0.1, SGD gives 1.95. For the first Adam step with zero moments, beta1=0.9 and beta2=0.999, m=0.05 and v=0.00025; bias-corrected values are 0.5 and 0.25. With epsilon=1e-8 the Adam result is approximately 1.900000002. It differs from SGD because Adam normalizes by the second-moment scale.\n\nAssumptions No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5.\n\n## Engineering example\n\nSynthetic first Adam step: θ = 1, gradient g = 2, zero initial moments, β₁ = 0.9, β₂ = 0.999 and learning rate 0.01 give m₁ = 0.2 and v₁ = 0.004. Bias correction gives m̂₁ = 2 and v̂₁ = 4; with ε = 10⁻⁸, θ becomes approximately 0.99.\n\nFor fixed predicted probabilities \\[0.2,0.6,0.8\\], a threshold of 0.5 gives decisions \\[0,1,1\\], while threshold 0.7 gives \\[0,0,1\\]. The weights and all three probabilities are unchanged. With labels \\[0,0,1\\], the second threshold removes one false positive on these three cases. This does not prove that 0.7 is better on future data; selecting it using the held-out test would also compromise that test’s independence.\n\nFor a scalar gradient g=0.5, weight 2 and learning rate 0.1, SGD gives 1.95. For the first Adam step with zero moments, beta1=0.9 and beta2=0.999, m=0.05 and v=0.00025; bias-corrected values are 0.5 and 0.25. With epsilon=1e-8 the Adam result is approximately 1.900000002. It differs from SGD because Adam normalizes by the second-moment scale.\n\n## Distinctions and tradeoffs\n\nBackpropagation computes derivatives; the optimizer decides the update.\n\nOptimizer state and saved activations consume substantial memory. Checkpointing can exchange extra computation for lower activation storage.\n\nLearning is an optimization process executed by physical arithmetic and memory systems.\n\n## Specifications and interfaces\n\nGradient\n\n∂loss / ∂weight\n\nLocal sensitivity of the objective.\n\nAdam state\n\nFirst and second moment estimates\n\nAdds storage beyond the weights.\n\n### Interfaces\n\nBatch and predictions Input\n\nObjective and activations\n\nModel parameters Output\n\nUpdated weight arrays\n\n## Check your understanding\n\n### Which process changes learned weights in a normal training step?\n\n1.  Displaying a response\n2.  Applying an optimizer update\n3.  Reading a KV cache\n\nAnswer and reasoning\n\nApplying an optimizer update\n\nThe optimizer updates parameters using gradients and its update rule.\n\n### A decision threshold changes from 0.5 to 0.7 with fixed model outputs. What necessarily changes?\n\n1.  The fitted weights\n2.  The training labels\n3.  Potential binary decisions, without requiring retraining\n\nAnswer and reasoning\n\nPotential binary decisions, without requiring retraining\n\nThresholding is a downstream decision rule; it can change labels without changing the fitted predictor.\n\n### Why is the first Adam change about 0.1 rather than 0.05?\n\n1.  Adam ignores gradients and always chooses random weights.\n2.  SGD and Adam use identical state equations.\n3.  The bias-corrected gradient is divided by its second-moment root.\n\nAnswer and reasoning\n\nThe bias-corrected gradient is divided by its second-moment root.\n\nThe bias-corrected gradient is divided by its second-moment root. No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5.\n\n## Historical context\n\n### 2014 · Adam combines momentum with adaptive scaling\n\nKingma and Ba released the Adam preprint in December 2014; its conference publication followed at ICLR 2015. The method maintains first and second gradient-moment estimates.\n\n[Kingma & Ba — Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980)\n\n## Evidence and scope\n\nThis follows the ordinary Adam equations in the linked paper. The gradient is supplied rather than derived here; clipping, weight decay, loss scaling and sharded optimizer state are excluded. Computing a gradient and applying an update remain separate steps.\n\nThe threshold arithmetic is authored. The existing sensor experience executes actual logistic updates, while these three probabilities are a separate worked decision example.\n\nMode: executable derived-gradient optimizer. No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [Kingma & Ba — Adam](https://arxiv.org/abs/1412.6980)\n-   [PyTorch — CrossEntropyLoss](https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html)\n-   [scikit-learn · Common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n-   [PyTorch Adam algorithm](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/loss-optimizer-75b6c32e6a4f283f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n\n### Related lessons\n\n-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)\n-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)\n-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)"
}