# Continue language modeling on a new domain Canonical: https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-continuation · Atlas v1.3.0 # Continue language modeling on a new domain Measure new-domain acquisition and old-domain retention from the same starting checkpoint. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#domain-adaptive-continuation)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveMeasure new-domain acquisition and old-domain retention from the same starting checkpoint. ## The mechanism Continued pretraining resumes parameter learning on a new distribution, often using the same language-model objective as the starting model. It differs from retrieval or prompting, which can change available context without changing weights. Domain adaptation research includes masked-language-model examples as well as causal decoder recipes; one should not label every cited experiment autoregressive. In the bounded lifecycle case, copy and swap use the same prompts with incompatible answers. This deliberately severe shift exposes the difference between learning a new rule and preserving an old one. 1. Save the base checkpoint and evaluate both domains before adaptation. 2. Draw only training examples from the declared new-domain mixture. 3. Retain or explicitly reset optimizer state according to the stage contract. 4. Measure both domain losses and answer accuracy at the same checkpoint after the fixed budget. ## Science and assumptions ### Continue language modeling on a new domain · worked relation $$ \Delta L_A=L_A(\theta_{\rm after})-L_A(\theta_{\rm before}),\qquad \Delta L_B=L_B(\theta_{\rm after})-L_B(\theta_{\rm before}) $$ Measure new-domain acquisition and old-domain retention from the same starting checkpoint. Symbols and units L\_A,L\_B Held-out losses under fixed domain-specific targets nats per target theta Checkpoint being evaluated parameters Delta L After-minus-before loss change nats per target For one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values. Assumptions Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. ## Engineering example For one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values. ## Distinctions and tradeoffs It is a mistake to conclude: The new domain necessarily improved. Concentrated adaptation can acquire a new distribution quickly but may trade away previous behavior. Retrieval has a different state and evaluation boundary. Adapt an existing checkpoint while measuring retained and acquired behavior. Measure new-domain acquisition and old-domain retention from the same starting checkpoint. ## Specifications and interfaces Learning mode executable continuation The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Measure new-domain acquisition and old-domain retention from the same starting checkpoint. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. ### Interfaces Inputs and state Input Save the base checkpoint and evaluate both domains before adaptation. Result and check Output Measure new-domain acquisition and old-domain retention from the same starting checkpoint. ## Check your understanding ### What does a positive old-domain loss change mean here? 1. The new domain necessarily improved. 2. No parameters changed. 3. The model assigns less likelihood to the old held-out targets. Answer and reasoning The model assigns less likelihood to the old held-out targets. The model assigns less likelihood to the old held-out targets. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. ## Evidence and scope Mode: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. ## Sources - [Don’t Stop Pretraining](https://aclanthology.org/2020.acl-main.740.pdf) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/domain-adaptive-continuation-1c23df0ce2faab3e.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/) - [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/) ### Related lessons - [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/) - [Retention, replay and transfer](https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/) - [Context extension is a learned-use test](https://semiaifoundry.com/learn/atlas/long-context-adaptation/) - [Compatible weight averaging and regressions](https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/)