# Loss, gradients & optimizer

Canonical: https://semiaifoundry.com/learn/atlas/loss-optimizer/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Learning · Atlas v1.3.0

# Loss, gradients & optimizer

Turn prediction errors into parameter updates.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#loss-optimizer)Logical structure · mapped onto physical hardware

Learning objectiveTrace one Adam update from a gradient through moment state to a changed parameter.

## The mechanism

A loss assigns a numerical training objective to the model’s predictions. Backpropagation applies the chain rule through the computation graph to obtain gradients. An optimizer uses those gradients and, often, additional state to update weights. Adam maintains estimates related to gradient moments. This backward computation and optimizer storage distinguish training from ordinary inference. Distributed execution must coordinate the appropriate gradients or shards before consistent updates can proceed.

A training objective, a decision threshold and an operational acceptance criterion answer different questions. Updating weights changes the predictive function; moving a threshold changes decisions for the same probabilities. Neither operation should be concealed inside an evaluation comparison. A frozen artifact and separate test set make these changes traceable into later deployment decisions.

An optimizer consumes a derived gradient; a supplied-gradient calculator teaches only the last part of that chain. The lifecycle experience adds an actual decoder loss and reverse pass before applying either SGD or Adam. Adam’s moments and step counter are learning state, not display history. Global norm clipping occurs before those moments. Changing the objective changes the gradient even if the optimizer setting is held fixed.

1.  Run the forward computation and evaluate the loss.
2.  Backpropagate derivatives through the recorded operations.
3.  Combine required distributed information and update parameters.
4.  The optimizer updates model parameters using the defined training loss and regularization.
5.  Evaluation keeps thresholds and accepted error tradeoffs explicit instead of claiming that lower loss necessarily produces a better application.
6.  Compute the gradient from the current batch loss and retain its denominator.
7.  Apply the declared global clipping rule.
8.  Update moments, bias-correct them and apply the chosen learning rate; save all resulting state.

## Science and assumptions

### Adam adapts updates using gradient moments

$$
\begin{aligned}g_t&=\nabla_\theta\mathcal L_t,\\m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t,\\v_t&=\beta_2v_{t-1}+(1-\beta_2)g_t^{\odot2},\\\hat m_t&=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t},\\\theta_{t+1}&=\theta_t-\eta\frac{\hat m_t}{\sqrt{\hat v_t}+\varepsilon}\end{aligned}
$$

Adam smooths gradients and their elementwise squares, corrects zero-initialization bias, then subtracts the step size times corrected first moment divided elementwise by square root of corrected second moment plus a stabilizer.

All squares, roots and divisions are elementwise. The two bias-correction relations are made explicit from the original assumptions. Parameter labels retain the source convention of updating θ\_t to θ\_(t+1).

Symbols and units

θₜ

Parameter vector before update t

parameter-coordinate units

ℒₜ

Step-t loss evaluated at the current parameters

loss units

gₜ

Gradient of step loss with respect to parameters

loss units per parameter-coordinate unit

mₜ,m̂ₜ

First moment and its zero-initialization bias correction

same units as gₜ

vₜ,v̂ₜ

Second raw moment and its bias correction

squared gradient units

β₁,β₂

Moment decay coefficients

fractions less than one

η

Step size applied to the normalized direction

parameter-coordinate units

ε

Positive denominator stabilizer

same units as square root of v̂ₜ

t

Positive update index for moment bias correction

integer index

⊙2

Elementwise square

operator

The first moment m smooths gradient direction; the second moment v tracks squared gradient magnitude. Bias-corrected estimates m̂ and v̂ scale each coordinate’s step. The optimizer consequently needs state arrays in addition to weights and gradients, connecting a mathematical update rule directly to accelerator memory requirements.

Assumptions Products, squares, and divisions are coordinate-wise. With zero initialization, m̂ₜ=mₜ/(1−β₁ᵗ) and v̂ₜ=vₜ/(1−β₂ᵗ). This is ordinary Adam; decoupled weight decay, clipping, loss scaling, and distributed sharding require additional rules.

### Compute gradients before applying the optimizer · worked relation

$$
m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,\quad\theta^+=\theta-\eta\frac{m_t/(1-\beta_1^t)}{\sqrt{v_t/(1-\beta_2^t)}+\epsilon}
$$

Verify the first actual gradient-driven SGD and Adam updates under explicit state.

Symbols and units

m\_t

First gradient moment

gradient units

v\_t

Second gradient moment

squared gradient units

g\_t

Gradient after declared clipping

gradient units

beta\_1,beta\_2

Moment decay factors

dimensionless

t

Optimizer update count

updates

eta

Learning rate

dimensionless

epsilon

Denominator stabilizer

gradient units

theta

Parameter

dimensionless

For a scalar gradient g=0.5, weight 2 and learning rate 0.1, SGD gives 1.95. For the first Adam step with zero moments, beta1=0.9 and beta2=0.999, m=0.05 and v=0.00025; bias-corrected values are 0.5 and 0.25. With epsilon=1e-8 the Adam result is approximately 1.900000002. It differs from SGD because Adam normalizes by the second-moment scale.

Assumptions No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5.

## Engineering example

Synthetic first Adam step: θ = 1, gradient g = 2, zero initial moments, β₁ = 0.9, β₂ = 0.999 and learning rate 0.01 give m₁ = 0.2 and v₁ = 0.004. Bias correction gives m̂₁ = 2 and v̂₁ = 4; with ε = 10⁻⁸, θ becomes approximately 0.99.

For fixed predicted probabilities \[0.2,0.6,0.8\], a threshold of 0.5 gives decisions \[0,1,1\], while threshold 0.7 gives \[0,0,1\]. The weights and all three probabilities are unchanged. With labels \[0,0,1\], the second threshold removes one false positive on these three cases. This does not prove that 0.7 is better on future data; selecting it using the held-out test would also compromise that test’s independence.

For a scalar gradient g=0.5, weight 2 and learning rate 0.1, SGD gives 1.95. For the first Adam step with zero moments, beta1=0.9 and beta2=0.999, m=0.05 and v=0.00025; bias-corrected values are 0.5 and 0.25. With epsilon=1e-8 the Adam result is approximately 1.900000002. It differs from SGD because Adam normalizes by the second-moment scale.

## Distinctions and tradeoffs

Backpropagation computes derivatives; the optimizer decides the update.

Optimizer state and saved activations consume substantial memory. Checkpointing can exchange extra computation for lower activation storage.

Learning is an optimization process executed by physical arithmetic and memory systems.

## Specifications and interfaces

Gradient

∂loss / ∂weight

Local sensitivity of the objective.

Adam state

First and second moment estimates

Adds storage beyond the weights.

### Interfaces

Batch and predictions Input

Objective and activations

Model parameters Output

Updated weight arrays

## Check your understanding

### Which process changes learned weights in a normal training step?

1.  Displaying a response
2.  Applying an optimizer update
3.  Reading a KV cache

Answer and reasoning

Applying an optimizer update

The optimizer updates parameters using gradients and its update rule.

### A decision threshold changes from 0.5 to 0.7 with fixed model outputs. What necessarily changes?

1.  The fitted weights
2.  The training labels
3.  Potential binary decisions, without requiring retraining

Answer and reasoning

Potential binary decisions, without requiring retraining

Thresholding is a downstream decision rule; it can change labels without changing the fitted predictor.

### Why is the first Adam change about 0.1 rather than 0.05?

1.  Adam ignores gradients and always chooses random weights.
2.  SGD and Adam use identical state equations.
3.  The bias-corrected gradient is divided by its second-moment root.

Answer and reasoning

The bias-corrected gradient is divided by its second-moment root.

The bias-corrected gradient is divided by its second-moment root. No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5.

## Historical context

### 2014 · Adam combines momentum with adaptive scaling

Kingma and Ba released the Adam preprint in December 2014; its conference publication followed at ICLR 2015. The method maintains first and second gradient-moment estimates.

[Kingma & Ba — Adam: A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980)

## Evidence and scope

This follows the ordinary Adam equations in the linked paper. The gradient is supplied rather than derived here; clipping, weight decay, loss scaling and sharded optimizer state are excluded. Computing a gradient and applying an update remain separate steps.

The threshold arithmetic is authored. The existing sensor experience executes actual logistic updates, while these three probabilities are a separate worked decision example.

Mode: executable derived-gradient optimizer. No weight decay or AMSGrad; elementwise squares/division. First-step worked values use an unclipped scalar gradient of 0.5. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [Kingma & Ba — Adam](https://arxiv.org/abs/1412.6980)
-   [PyTorch — CrossEntropyLoss](https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html)
-   [scikit-learn · Common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)
-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)
-   [PyTorch Adam algorithm](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/loss-optimizer-75b6c32e6a4f283f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)
-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)

### Related lessons

-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)
-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)
-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)
-   [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/)
-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)
-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)
-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)
-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)