{"id":"checkpoint-averaging-boundaries","name":"Compatible weight averaging and regressions","shortName":"Compatible weight averaging and regressions","role":"Check parameter alignment before averaging and evaluate the resulting function.","description":"Weight averaging combines numbers at matching parameter coordinates. Shape equality is necessary but does not prove semantic alignment: hidden units can be permuted while preserving the original function. Independently trained models can therefore disagree about which coordinate represents which feature. A shared initialization and compatible training recipe make an average more interpretable, but do not guarantee that every task is preserved. An averaged checkpoint also needs an explicit optimizer policy; averaging weights does not establish suitable Adam moments or a resume cursor.","mechanism":["Check architecture, tokenizer, parameter order and source checkpoint identities.","Form a declared weighted average only across compatible coordinates.","Evaluate both original checkpoints and the average on identical reserved cases.","Treat averaged weights as a new model identity and explicitly choose fresh or otherwise justified optimizer state."],"learningObjective":"Check parameter alignment before averaging and evaluate the resulting function.","engineeringExample":"Two equivalent two-unit linear networks represent y=x with incoming vector [1,−1] and outgoing [0.5,−0.5], or with both vectors permuted to [−1,1] and [−0.5,0.5]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.","evidenceNotes":"Mode: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Check parameter alignment before averaging and evaluate the resulting function.","context":"Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Check architecture, tokenizer, parameter order and source checkpoint identities."},{"name":"Result and check","direction":"out","signal":"Check parameter alignment before averaging and evaluate the resulting function."}],"tradeoff":"Averaging may combine nearby solutions without storing an ensemble at inference, but it can also create a poor function and ambiguous optimizer history.","whyAI":"Adapt an existing checkpoint while measuring retained and acquired behavior. Check parameter alignment before averaging and evaluate the resulting function.","misconception":"It is a mistake to conclude: Equal shapes imply identical learned features.","check":{"question":"Why do equal tensor shapes fail to guarantee a useful average?","options":["Equal shapes imply identical learned features.","Averaging always preserves each source prediction.","Equivalent functions can use different hidden-unit coordinate assignments."],"answer":2,"explanation":"Equivalent functions can use different hidden-unit coordinate assignments. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core."},"prerequisites":["checkpoint","evaluation"],"related":["checkpoint","evaluation","model-lifecycle-stage-contract","domain-adaptive-continuation","replay-retention-forgetting","long-context-adaptation"],"sources":[{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-continuation","subbranch":"Compatible weight averaging and regressions","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.","science":[{"title":"Compatible weight averaging and regressions · worked relation","equation":"\\bar\\theta=\\sum_k a_k\\theta_k,\\qquad \\sum_k a_k=1,\\quad a_k\\ge0","explanation":"Two equivalent two-unit linear networks represent y=x with incoming vector [1,−1] and outgoing [0.5,−0.5], or with both vectors permuted to [−1,1] and [−0.5,0.5]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.","assumptions":"Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core."}],"history":[]}