The mechanism
The derivative specifies a local direction; the optimizer and schedule specify the actual state transition. Global norm clipping rescales an entire gradient vector before the update. Adam additionally carries first and second moments and corrects their initialization bias. AdamW applies a distinct shrinkage term rather than feeding an L2 derivative into those adaptive moments. A schedule changes the step size over updates, and its position belongs in a restart record. The lifecycle core executes clipping and constant-rate Adam or SGD; warmup, decay schedules and AdamW below are worked extensions, not hidden features of that core.
- Compute the raw gradient norm before any clipping.
- Apply one declared global scale to all gradient entries.
- Update optimizer moments and their time counter from the clipped gradient.
- Apply the stated learning rate and, only when selected by the algorithm, a separate decay term; compare fixed-seed runs.
Science and assumptions
Initialization, learning rate and stable updates · worked relation
Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.
Symbols and units
- g,g_c
- Raw and globally clipped gradient vectors
loss per parameter
- c
- Gradient norm limit
loss per parameter
- theta
- Parameter vector
dimensionless
- eta
- Step size in the declared numerical convention
dimensionless
- u
- Adaptive update direction
dimensionless
- lambda
- Decoupled decay coefficient
dimensionless
For raw gradient [3,4], norm is 5. A clip threshold of 1 gives [0.6,0.8]. With initial weights [2,2] and SGD rate 0.1, the result is [1.94,1.92]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.
Assumptions For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.
Engineering example
For raw gradient [3,4], norm is 5. A clip threshold of 1 gives [0.6,0.8]. With initial weights [2,2] and SGD rate 0.1, the result is [1.94,1.92]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.
Distinctions and tradeoffs
It is a mistake to conclude: Clamping each entry independently to 1.
Clipping can contain extreme updates but can also suppress useful signal. Stability on a small run does not determine a universally good schedule.
Turn an objective into a stable, restartable experiment with controlled budgets. Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.
Specifications and interfaces
- Learning mode
- executable clipping; worked schedule and AdamW
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.
For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.
Interfaces
- Inputs and state Input
- Compute the raw gradient norm before any clipping.
- Result and check Output
- Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.
Check your understanding
What preserves the direction of gradient [3,4] during norm clipping?
- Clamping each entry independently to 1.
- Replacing the gradient with [1,0].
- Multiplying both entries by 1/5.
Answer and reasoning
Multiplying both entries by 1/5.
Multiplying both entries by 1/5. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.
Evidence and scope
Mode: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.