{
  "schemaVersion": 1,
  "id": "atlas-lesson-domain-adaptive-continuation",
  "title": "Continue language modeling on a new domain",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Measure new-domain acquisition and old-domain retention from the same starting checkpoint.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-continuation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/domain-adaptive-continuation-1c23df0ce2faab3e.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#domain-adaptive-continuation",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ee0071ff85f21ecf3b9892297b027f6989c1ec5613a8848535367169b4f50305",
  "contentLength": 6381,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-domain-adaptive-continuation.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-domain-adaptive-continuation.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Continue language modeling on a new domain",
      "start": 45,
      "end": 462
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 462,
      "end": 1410
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1410,
      "end": 1438
    },
    {
      "id": "section-4",
      "title": "Continue language modeling on a new domain · worked relation",
      "start": 1438,
      "end": 2557
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2557,
      "end": 3010
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3010,
      "end": 3436
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3436,
      "end": 3900
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3900,
      "end": 4128
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4128,
      "end": 4157
    },
    {
      "id": "section-10",
      "title": "What does a positive old-domain loss change mean here?",
      "start": 4157,
      "end": 4683
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4683,
      "end": 5352
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5352,
      "end": 5649
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5649,
      "end": 5672
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5672,
      "end": 5906
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5906,
      "end": 6381
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#domain-adaptive-continuation"
    },
    {
      "title": "Don’t Stop Pretraining",
      "url": "https://aclanthology.org/2020.acl-main.740.pdf"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/domain-adaptive-continuation-1c23df0ce2faab3e.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Describe a stage by its changed contract",
      "url": "https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/"
    },
    {
      "title": "Retention, replay and transfer",
      "url": "https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/"
    },
    {
      "title": "Context extension is a learned-use test",
      "url": "https://semiaifoundry.com/learn/atlas/long-context-adaptation/"
    },
    {
      "title": "Compatible weight averaging and regressions",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/"
    }
  ],
  "recordHash": "524bc269948c1fabd47152e95588eff66b03966204d9e4c4cf6726d5d8d8690b",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-continuation · Atlas v1.3.0\n\n# Continue language modeling on a new domain\n\nMeasure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#domain-adaptive-continuation)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveMeasure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\n## The mechanism\n\nContinued pretraining resumes parameter learning on a new distribution, often using the same language-model objective as the starting model. It differs from retrieval or prompting, which can change available context without changing weights. Domain adaptation research includes masked-language-model examples as well as causal decoder recipes; one should not label every cited experiment autoregressive. In the bounded lifecycle case, copy and swap use the same prompts with incompatible answers. This deliberately severe shift exposes the difference between learning a new rule and preserving an old one.\n\n1.  Save the base checkpoint and evaluate both domains before adaptation.\n2.  Draw only training examples from the declared new-domain mixture.\n3.  Retain or explicitly reset optimizer state according to the stage contract.\n4.  Measure both domain losses and answer accuracy at the same checkpoint after the fixed budget.\n\n## Science and assumptions\n\n### Continue language modeling on a new domain · worked relation\n\n$$\n\\Delta L_A=L_A(\\theta_{\\rm after})-L_A(\\theta_{\\rm before}),\\qquad \\Delta L_B=L_B(\\theta_{\\rm after})-L_B(\\theta_{\\rm before})\n$$\n\nMeasure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\nSymbols and units\n\nL\\_A,L\\_B\n\nHeld-out losses under fixed domain-specific targets\n\nnats per target\n\ntheta\n\nCheckpoint being evaluated\n\nparameters\n\nDelta L\n\nAfter-minus-before loss change\n\nnats per target\n\nFor one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values.\n\nAssumptions Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.\n\n## Engineering example\n\nFor one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: The new domain necessarily improved.\n\nConcentrated adaptation can acquire a new distribution quickly but may trade away previous behavior. Retrieval has a different state and evaluation boundary.\n\nAdapt an existing checkpoint while measuring retained and acquired behavior. Measure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable continuation\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nMeasure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\nSame evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.\n\n### Interfaces\n\nInputs and state Input\n\nSave the base checkpoint and evaluate both domains before adaptation.\n\nResult and check Output\n\nMeasure new-domain acquisition and old-domain retention from the same starting checkpoint.\n\n## Check your understanding\n\n### What does a positive old-domain loss change mean here?\n\n1.  The new domain necessarily improved.\n2.  No parameters changed.\n3.  The model assigns less likelihood to the old held-out targets.\n\nAnswer and reasoning\n\nThe model assigns less likelihood to the old held-out targets.\n\nThe model assigns less likelihood to the old held-out targets. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.\n\n## Evidence and scope\n\nMode: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.\n\n## Sources\n\n-   [Don’t Stop Pretraining](https://aclanthology.org/2020.acl-main.740.pdf)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/domain-adaptive-continuation-1c23df0ce2faab3e.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n\n### Related lessons\n\n-   [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/)\n-   [Retention, replay and transfer](https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/)\n-   [Context extension is a learned-use test](https://semiaifoundry.com/learn/atlas/long-context-adaptation/)\n-   [Compatible weight averaging and regressions](https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/)"
}