The mechanism
Continued pretraining resumes parameter learning on a new distribution, often using the same language-model objective as the starting model. It differs from retrieval or prompting, which can change available context without changing weights. Domain adaptation research includes masked-language-model examples as well as causal decoder recipes; one should not label every cited experiment autoregressive. In the bounded lifecycle case, copy and swap use the same prompts with incompatible answers. This deliberately severe shift exposes the difference between learning a new rule and preserving an old one.
- Save the base checkpoint and evaluate both domains before adaptation.
- Draw only training examples from the declared new-domain mixture.
- Retain or explicitly reset optimizer state according to the stage contract.
- Measure both domain losses and answer accuracy at the same checkpoint after the fixed budget.
Science and assumptions
Continue language modeling on a new domain · worked relation
Measure new-domain acquisition and old-domain retention from the same starting checkpoint.
Symbols and units
- L_A,L_B
- Held-out losses under fixed domain-specific targets
nats per target
- theta
- Checkpoint being evaluated
parameters
- Delta L
- After-minus-before loss change
nats per target
For one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values.
Assumptions Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.
Engineering example
For one binary prompt, suppose a model assigns probability 0.9 to the old answer and 0.1 to the new answer. Old/new NLLs are 0.105361 and 2.302585 nats. If adaptation changes these probabilities to 0.2 and 0.8, the losses become 1.609438 and 0.223144. New-domain loss improves by 2.079442 while old-domain loss worsens by 1.504077. These probabilities are an authored worked example; the studio reports its own computed values.
Distinctions and tradeoffs
It is a mistake to conclude: The new domain necessarily improved.
Concentrated adaptation can acquire a new distribution quickly but may trade away previous behavior. Retrieval has a different state and evaluation boundary.
Adapt an existing checkpoint while measuring retained and acquired behavior. Measure new-domain acquisition and old-domain retention from the same starting checkpoint.
Specifications and interfaces
- Learning mode
- executable continuation
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Measure new-domain acquisition and old-domain retention from the same starting checkpoint.
Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.
Interfaces
- Inputs and state Input
- Save the base checkpoint and evaluate both domains before adaptation.
- Result and check Output
- Measure new-domain acquisition and old-domain retention from the same starting checkpoint.
Check your understanding
What does a positive old-domain loss change mean here?
- The new domain necessarily improved.
- No parameters changed.
- The model assigns less likelihood to the old held-out targets.
Answer and reasoning
The model assigns less likelihood to the old held-out targets.
The model assigns less likelihood to the old held-out targets. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.
Evidence and scope
Mode: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: executable continuation. Same evaluation cases, masks and tokenizer before/after; negative loss change is improvement. Binary illustrative probabilities do not describe every vocabulary class in the core.