# Compatible weight averaging and regressions Canonical: https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-continuation · Atlas v1.3.0 # Compatible weight averaging and regressions Check parameter alignment before averaging and evaluate the resulting function. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#checkpoint-averaging-boundaries)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCheck parameter alignment before averaging and evaluate the resulting function. ## The mechanism Weight averaging combines numbers at matching parameter coordinates. Shape equality is necessary but does not prove semantic alignment: hidden units can be permuted while preserving the original function. Independently trained models can therefore disagree about which coordinate represents which feature. A shared initialization and compatible training recipe make an average more interpretable, but do not guarantee that every task is preserved. An averaged checkpoint also needs an explicit optimizer policy; averaging weights does not establish suitable Adam moments or a resume cursor. 1. Check architecture, tokenizer, parameter order and source checkpoint identities. 2. Form a declared weighted average only across compatible coordinates. 3. Evaluate both original checkpoints and the average on identical reserved cases. 4. Treat averaged weights as a new model identity and explicitly choose fresh or otherwise justified optimizer state. ## Science and assumptions ### Compatible weight averaging and regressions · worked relation $$ \bar\theta=\sum_k a_k\theta_k,\qquad \sum_k a_k=1,\quad a_k\ge0 $$ Check parameter alignment before averaging and evaluate the resulting function. Symbols and units theta\_k Coordinate-aligned candidate weights parameters a\_k Averaging weights dimensionless bar theta New averaged checkpoint parameters Two equivalent two-unit linear networks represent y=x with incoming vector \[1,−1\] and outgoing \[0.5,−0.5\], or with both vectors permuted to \[−1,1\] and \[−0.5,0.5\]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality. Assumptions Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. ## Engineering example Two equivalent two-unit linear networks represent y=x with incoming vector \[1,−1\] and outgoing \[0.5,−0.5\], or with both vectors permuted to \[−1,1\] and \[−0.5,0.5\]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality. ## Distinctions and tradeoffs It is a mistake to conclude: Equal shapes imply identical learned features. Averaging may combine nearby solutions without storing an ensemble at inference, but it can also create a poor function and ambiguous optimizer history. Adapt an existing checkpoint while measuring retained and acquired behavior. Check parameter alignment before averaging and evaluate the resulting function. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Check parameter alignment before averaging and evaluate the resulting function. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. ### Interfaces Inputs and state Input Check architecture, tokenizer, parameter order and source checkpoint identities. Result and check Output Check parameter alignment before averaging and evaluate the resulting function. ## Check your understanding ### Why do equal tensor shapes fail to guarantee a useful average? 1. Equal shapes imply identical learned features. 2. Averaging always preserves each source prediction. 3. Equivalent functions can use different hidden-unit coordinate assignments. Answer and reasoning Equivalent functions can use different hidden-unit coordinate assignments. Equivalent functions can use different hidden-unit coordinate assignments. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. ## Evidence and scope Mode: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. ## Sources - [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-averaging-boundaries-6f89c616c7334d7b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/) - [Continue language modeling on a new domain](https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/) - [Retention, replay and transfer](https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/) - [Context extension is a learned-use test](https://semiaifoundry.com/learn/atlas/long-context-adaptation/)