{"id":"optimizer-schedule-stability","name":"Initialization, learning rate and stable updates","shortName":"Initialization, learning rate and stable updates","role":"Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.","description":"The derivative specifies a local direction; the optimizer and schedule specify the actual state transition. Global norm clipping rescales an entire gradient vector before the update. Adam additionally carries first and second moments and corrects their initialization bias. AdamW applies a distinct shrinkage term rather than feeding an L2 derivative into those adaptive moments. A schedule changes the step size over updates, and its position belongs in a restart record. The lifecycle core executes clipping and constant-rate Adam or SGD; warmup, decay schedules and AdamW below are worked extensions, not hidden features of that core.","mechanism":["Compute the raw gradient norm before any clipping.","Apply one declared global scale to all gradient entries.","Update optimizer moments and their time counter from the clipped gradient.","Apply the stated learning rate and, only when selected by the algorithm, a separate decay term; compare fixed-seed runs."],"learningObjective":"Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.","engineeringExample":"For raw gradient [3,4], norm is 5. A clip threshold of 1 gives [0.6,0.8]. With initial weights [2,2] and SGD rate 0.1, the result is [1.94,1.92]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.","evidenceNotes":"Mode: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable clipping; worked schedule and AdamW","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.","context":"For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Compute the raw gradient norm before any clipping."},{"name":"Result and check","direction":"out","signal":"Separate gradient clipping, adaptive moments, decay and the learning-rate schedule."}],"tradeoff":"Clipping can contain extreme updates but can also suppress useful signal. Stability on a small run does not determine a universally good schedule.","whyAI":"Turn an objective into a stable, restartable experiment with controlled budgets. Separate gradient clipping, adaptive moments, decay and the learning-rate schedule.","misconception":"It is a mistake to conclude: Clamping each entry independently to 1.","check":{"question":"What preserves the direction of gradient [3,4] during norm clipping?","options":["Clamping each entry independently to 1.","Replacing the gradient with [1,0].","Multiplying both entries by 1/5."],"answer":2,"explanation":"Multiplying both entries by 1/5. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule."},"prerequisites":["loss-optimizer","residual-normalization"],"related":["loss-optimizer","residual-normalization","causal-pretraining-objective","microbatch-equivalent-updates"],"sources":[{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"},{"title":"Decoupled Weight Decay Regularization","url":"https://arxiv.org/html/1711.05101v3"},{"title":"PyTorch Adam algorithm","url":"https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-pretraining","subbranch":"Initialization, learning rate and stable updates","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable clipping; worked schedule and AdamW. For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule.","science":[{"title":"Initialization, learning rate and stable updates · worked relation","equation":"g_c=g\\min\\!\\left(1,\\frac{c}{\\lVert g\\rVert_2}\\right),\\qquad \\theta^+=\\theta-\\eta u-\\eta\\lambda\\theta","explanation":"For raw gradient [3,4], norm is 5. A clip threshold of 1 gives [0.6,0.8]. With initial weights [2,2] and SGD rate 0.1, the result is [1.94,1.92]. In a separate scalar AdamW example with weight 2, adaptive direction 1, learning rate 0.1 and decay coefficient 0.01, the result is 2−0.1×1−0.1×0.01×2=1.898. That example uses the common learning-rate-scaled decay convention explicitly.","assumptions":"For zero gradient use clipping multiplier one. The second equation declares one AdamW convention; lifecycle core sets decay to zero and has a constant schedule."}],"history":[]}