The mechanism
Weight averaging combines numbers at matching parameter coordinates. Shape equality is necessary but does not prove semantic alignment: hidden units can be permuted while preserving the original function. Independently trained models can therefore disagree about which coordinate represents which feature. A shared initialization and compatible training recipe make an average more interpretable, but do not guarantee that every task is preserved. An averaged checkpoint also needs an explicit optimizer policy; averaging weights does not establish suitable Adam moments or a resume cursor.
- Check architecture, tokenizer, parameter order and source checkpoint identities.
- Form a declared weighted average only across compatible coordinates.
- Evaluate both original checkpoints and the average on identical reserved cases.
- Treat averaged weights as a new model identity and explicitly choose fresh or otherwise justified optimizer state.
Science and assumptions
Compatible weight averaging and regressions · worked relation
Check parameter alignment before averaging and evaluate the resulting function.
Symbols and units
- theta_k
- Coordinate-aligned candidate weights
parameters
- a_k
- Averaging weights
dimensionless
- bar theta
- New averaged checkpoint
parameters
Two equivalent two-unit linear networks represent y=x with incoming vector [1,−1] and outgoing [0.5,−0.5], or with both vectors permuted to [−1,1] and [−0.5,0.5]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.
Assumptions Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.
Engineering example
Two equivalent two-unit linear networks represent y=x with incoming vector [1,−1] and outgoing [0.5,−0.5], or with both vectors permuted to [−1,1] and [−0.5,0.5]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.
Distinctions and tradeoffs
It is a mistake to conclude: Equal shapes imply identical learned features.
Averaging may combine nearby solutions without storing an ensemble at inference, but it can also create a poor function and ambiguous optimizer history.
Adapt an existing checkpoint while measuring retained and acquired behavior. Check parameter alignment before averaging and evaluate the resulting function.
Specifications and interfaces
- Learning mode
- worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Check parameter alignment before averaging and evaluate the resulting function.
Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.
Interfaces
- Inputs and state Input
- Check architecture, tokenizer, parameter order and source checkpoint identities.
- Result and check Output
- Check parameter alignment before averaging and evaluate the resulting function.
Check your understanding
Why do equal tensor shapes fail to guarantee a useful average?
- Equal shapes imply identical learned features.
- Averaging always preserves each source prediction.
- Equivalent functions can use different hidden-unit coordinate assignments.
Answer and reasoning
Equivalent functions can use different hidden-unit coordinate assignments.
Equivalent functions can use different hidden-unit coordinate assignments. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.
Evidence and scope
Mode: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.