{
  "schemaVersion": 1,
  "id": "atlas-lesson-checkpoint-averaging-boundaries",
  "title": "Compatible weight averaging and regressions",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Check parameter alignment before averaging and evaluate the resulting function.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-continuation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-averaging-boundaries-6f89c616c7334d7b.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#checkpoint-averaging-boundaries",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "9b04028466aabf214dc6611a12e0da892b0dfe46c8f845b5cd7ed033ab80381f",
  "contentLength": 6198,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint-averaging-boundaries.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint-averaging-boundaries.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Compatible weight averaging and regressions",
      "start": 45,
      "end": 444
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 444,
      "end": 1416
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1416,
      "end": 1444
    },
    {
      "id": "section-4",
      "title": "Compatible weight averaging and regressions · worked relation",
      "start": 1444,
      "end": 2436
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2436,
      "end": 2885
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2885,
      "end": 3305
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3305,
      "end": 3737
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3737,
      "end": 3965
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3965,
      "end": 3994
    },
    {
      "id": "section-10",
      "title": "Why do equal tensor shapes fail to guarantee a useful average?",
      "start": 3994,
      "end": 4590
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4590,
      "end": 5217
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5217,
      "end": 5500
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5500,
      "end": 5523
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5523,
      "end": 5727
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5727,
      "end": 6198
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#checkpoint-averaging-boundaries"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-averaging-boundaries-6f89c616c7334d7b.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Describe a stage by its changed contract",
      "url": "https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/"
    },
    {
      "title": "Continue language modeling on a new domain",
      "url": "https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/"
    },
    {
      "title": "Retention, replay and transfer",
      "url": "https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/"
    },
    {
      "title": "Context extension is a learned-use test",
      "url": "https://semiaifoundry.com/learn/atlas/long-context-adaptation/"
    }
  ],
  "recordHash": "a3d22b6d4026cbd548e3ee66dfce9ae782bcbd0fcfb92b688f08a18cbcf1fe14",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-continuation · Atlas v1.3.0\n\n# Compatible weight averaging and regressions\n\nCheck parameter alignment before averaging and evaluate the resulting function.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#checkpoint-averaging-boundaries)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCheck parameter alignment before averaging and evaluate the resulting function.\n\n## The mechanism\n\nWeight averaging combines numbers at matching parameter coordinates. Shape equality is necessary but does not prove semantic alignment: hidden units can be permuted while preserving the original function. Independently trained models can therefore disagree about which coordinate represents which feature. A shared initialization and compatible training recipe make an average more interpretable, but do not guarantee that every task is preserved. An averaged checkpoint also needs an explicit optimizer policy; averaging weights does not establish suitable Adam moments or a resume cursor.\n\n1.  Check architecture, tokenizer, parameter order and source checkpoint identities.\n2.  Form a declared weighted average only across compatible coordinates.\n3.  Evaluate both original checkpoints and the average on identical reserved cases.\n4.  Treat averaged weights as a new model identity and explicitly choose fresh or otherwise justified optimizer state.\n\n## Science and assumptions\n\n### Compatible weight averaging and regressions · worked relation\n\n$$\n\\bar\\theta=\\sum_k a_k\\theta_k,\\qquad \\sum_k a_k=1,\\quad a_k\\ge0\n$$\n\nCheck parameter alignment before averaging and evaluate the resulting function.\n\nSymbols and units\n\ntheta\\_k\n\nCoordinate-aligned candidate weights\n\nparameters\n\na\\_k\n\nAveraging weights\n\ndimensionless\n\nbar theta\n\nNew averaged checkpoint\n\nparameters\n\nTwo equivalent two-unit linear networks represent y=x with incoming vector \\[1,−1\\] and outgoing \\[0.5,−0.5\\], or with both vectors permuted to \\[−1,1\\] and \\[−0.5,0.5\\]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.\n\nAssumptions Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.\n\n## Engineering example\n\nTwo equivalent two-unit linear networks represent y=x with incoming vector \\[1,−1\\] and outgoing \\[0.5,−0.5\\], or with both vectors permuted to \\[−1,1\\] and \\[−0.5,0.5\\]. Each dot product is 1. Averaging corresponding incoming and outgoing entries gives zeros, so the averaged network produces y=0. By contrast, averaging compatible scalar weights 1 and 3 gives 2; that arithmetic alone still does not predict task quality.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Equal shapes imply identical learned features.\n\nAveraging may combine nearby solutions without storing an ensemble at inference, but it can also create a poor function and ambiguous optimizer history.\n\nAdapt an existing checkpoint while measuring retained and acquired behavior. Check parameter alignment before averaging and evaluate the resulting function.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCheck parameter alignment before averaging and evaluate the resulting function.\n\nCompatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.\n\n### Interfaces\n\nInputs and state Input\n\nCheck architecture, tokenizer, parameter order and source checkpoint identities.\n\nResult and check Output\n\nCheck parameter alignment before averaging and evaluate the resulting function.\n\n## Check your understanding\n\n### Why do equal tensor shapes fail to guarantee a useful average?\n\n1.  Equal shapes imply identical learned features.\n2.  Averaging always preserves each source prediction.\n3.  Equivalent functions can use different hidden-unit coordinate assignments.\n\nAnswer and reasoning\n\nEquivalent functions can use different hidden-unit coordinate assignments.\n\nEquivalent functions can use different hidden-unit coordinate assignments. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.\n\n## Evidence and scope\n\nMode: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Compatible parameter coordinates and tokenizer required. Permutation counterexample is an exact linear network; no averaging feature is executed by the lifecycle core.\n\n## Sources\n\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-averaging-boundaries-6f89c616c7334d7b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/)\n-   [Continue language modeling on a new domain](https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/)\n-   [Retention, replay and transfer](https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/)\n-   [Context extension is a learned-use test](https://semiaifoundry.com/learn/atlas/long-context-adaptation/)"
}