{
  "schemaVersion": 1,
  "id": "atlas-lesson-replay-retention-forgetting",
  "title": "Retention, replay and transfer",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Reason about a fixed-budget retention tradeoff without treating replay as a guarantee.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/replay-retention-forgetting/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-continuation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable replay; worked ideal optimum. One identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/replay-retention-forgetting-15a494e0da3c020a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#replay-retention-forgetting",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "dcda482c0744abb8e683df3d0f25a4df4dc85eec4edb440b23cff3b728eaf5e1",
  "contentLength": 6219,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-replay-retention-forgetting.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-replay-retention-forgetting.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Retention, replay and transfer",
      "start": 45,
      "end": 441
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 441,
      "end": 1415
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1415,
      "end": 1443
    },
    {
      "id": "section-4",
      "title": "Retention, replay and transfer · worked relation",
      "start": 1443,
      "end": 2462
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2462,
      "end": 2868
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2868,
      "end": 3289
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3289,
      "end": 3759
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3759,
      "end": 3983
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3983,
      "end": 4012
    },
    {
      "id": "section-10",
      "title": "Why can replay improve old-answer likelihood without restoring old-answer accuracy?",
      "start": 4012,
      "end": 4645
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4645,
      "end": 5334
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5334,
      "end": 5690
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5690,
      "end": 5713
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5713,
      "end": 5849
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5849,
      "end": 6219
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#replay-retention-forgetting"
    },
    {
      "title": "Don’t Stop Pretraining",
      "url": "https://aclanthology.org/2020.acl-main.740.pdf"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/replay-retention-forgetting-15a494e0da3c020a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Continue language modeling on a new domain",
      "url": "https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/"
    },
    {
      "title": "Describe a stage by its changed contract",
      "url": "https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/"
    },
    {
      "title": "Context extension is a learned-use test",
      "url": "https://semiaifoundry.com/learn/atlas/long-context-adaptation/"
    },
    {
      "title": "Compatible weight averaging and regressions",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/"
    }
  ],
  "recordHash": "4f77e5e2707fc5339ced3eccdbf6e7dd9161003ffe6cd043baed4c29061cd011",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-continuation · Atlas v1.3.0\n\n# Retention, replay and transfer\n\nReason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#replay-retention-forgetting)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveReason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\n## The mechanism\n\nReplay mixes old training examples into adaptation. It changes the expected objective and consumes part of the same update budget. Comparing a longer replay run with a shorter new-only run confounds mixture with computation. Retention must be evaluated on reserved old-domain examples, not merely on repeated training items. In the lifecycle task the same prompt has opposite desired answers in copy and swap. No deterministic single-answer policy without a domain identifier can satisfy both simultaneously. That impossibility is a property of this constructed task, not a universal claim about continual learning.\n\n1.  Hold the starting checkpoint, total draws and optimizer policy fixed.\n2.  Select a declared old-domain replay probability.\n3.  Record realized domain draws rather than assuming the expected fraction occurred exactly.\n4.  Compare both reserved-domain losses, and explain whether conflict or insufficient acquisition limits the result.\n\n## Science and assumptions\n\n### Retention, replay and transfer · worked relation\n\n$$\nL(p)=-q\\log p-(1-q)\\log(1-p),\\qquad p^*=q\\quad(0<q<1)\n$$\n\nReason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\nSymbols and units\n\np\n\nProbability of the old-domain answer\n\ndimensionless\n\nq\n\nOld-domain mixture weight\n\ndimensionless\n\nL\n\nExpected binary target loss\n\nnats per target\n\np\\*\n\nOptimal probability for the specified conflicting prompt\n\ndimensionless\n\nFor one conflicting binary prompt let p be the old-answer probability and q the old-example mixture fraction. Expected NLL is −q ln p−(1−q)ln(1−p), minimized at p=q. With q=0.25, old/new losses at that optimum are ln4=1.386294 and −ln0.75=0.287682 nats. Greedy output selects the new answer, so old accuracy is zero even though replay has raised old-answer probability above zero.\n\nAssumptions One identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum.\n\n## Engineering example\n\nFor one conflicting binary prompt let p be the old-answer probability and q the old-example mixture fraction. Expected NLL is −q ln p−(1−q)ln(1−p), minimized at p=q. With q=0.25, old/new losses at that optimum are ln4=1.386294 and −ln0.75=0.287682 nats. Greedy output selects the new answer, so old accuracy is zero even though replay has raised old-answer probability above zero.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Likelihood and accuracy are the same statistic.\n\nReplay allocates scarce updates to retention; it can reduce new-task progress and cannot solve an information-theoretic label conflict by itself.\n\nAdapt an existing checkpoint while measuring retained and acquired behavior. Reason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable replay; worked ideal optimum\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nReason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\nOne identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum.\n\n### Interfaces\n\nInputs and state Input\n\nHold the starting checkpoint, total draws and optimizer policy fixed.\n\nResult and check Output\n\nReason about a fixed-budget retention tradeoff without treating replay as a guarantee.\n\n## Check your understanding\n\n### Why can replay improve old-answer likelihood without restoring old-answer accuracy?\n\n1.  Probability can rise while remaining below the greedy decision threshold.\n2.  Likelihood and accuracy are the same statistic.\n3.  Replay guarantees both incompatible labels are always correct.\n\nAnswer and reasoning\n\nProbability can rise while remaining below the greedy decision threshold.\n\nProbability can rise while remaining below the greedy decision threshold. One identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum.\n\n## Evidence and scope\n\nMode: executable replay; worked ideal optimum. One identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable replay; worked ideal optimum. One identical prompt with contradictory binary labels, unlimited fitting of its probability and no domain tag. Finite decoder optimization need not reach this ideal optimum.\n\n## Sources\n\n-   [Don’t Stop Pretraining](https://aclanthology.org/2020.acl-main.740.pdf)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/replay-retention-forgetting-15a494e0da3c020a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Continue language modeling on a new domain](https://semiaifoundry.com/learn/atlas/domain-adaptive-continuation/)\n\n### Related lessons\n\n-   [Describe a stage by its changed contract](https://semiaifoundry.com/learn/atlas/model-lifecycle-stage-contract/)\n-   [Context extension is a learned-use test](https://semiaifoundry.com/learn/atlas/long-context-adaptation/)\n-   [Compatible weight averaging and regressions](https://semiaifoundry.com/learn/atlas/checkpoint-averaging-boundaries/)"
}