{
  "schemaVersion": 1,
  "id": "atlas-lesson-checkpoint-retained-work",
  "title": "Checkpointing, failure and retained useful work",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Balances checkpoint overhead against lost progress and recovery delay.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Data-center operations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Failure rates and times are synthetic. Provider-reported speedups are not imported as defaults; real failure correlation and checkpoint durability require operational evidence.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-retained-work-34f5856318637549.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#checkpoint-retained-work",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "6bbd59bce61dcbf90373833a68950babeec1c66c495a21b0dbe80b2b83b272c3",
  "contentLength": 5042,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint-retained-work.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint-retained-work.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Checkpointing, failure and retained useful work",
      "start": 39,
      "end": 398
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 398,
      "end": 1548
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1548,
      "end": 1576
    },
    {
      "id": "section-4",
      "title": "Simple checkpoint loss fraction",
      "start": 1576,
      "end": 2428
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2428,
      "end": 2659
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2659,
      "end": 3067
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3067,
      "end": 3330
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3330,
      "end": 3514
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3514,
      "end": 3543
    },
    {
      "id": "section-10",
      "title": "What should a useful-work metric exclude after a restart?",
      "start": 3543,
      "end": 3911
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3911,
      "end": 4258
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4258,
      "end": 4668
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4668,
      "end": 4691
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4691,
      "end": 4903
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4903,
      "end": 5042
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#checkpoint-retained-work"
    },
    {
      "title": "Elastic training and optimized checkpointing improve ML Goodput",
      "url": "https://cloud.google.com/blog/products/ai-machine-learning/elastic-training-and-optimized-checkpointing-improve-ml-goodput"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-retained-work-34f5856318637549.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "Storage paths for datasets and checkpoints",
      "url": "https://semiaifoundry.com/learn/atlas/storage-input-checkpoint-path/"
    }
  ],
  "recordHash": "99c02d108a1eea5277477a2d30f3ae87a311cca9f63c919a74d12dfff9e95ae1",
  "contentFormat": "text/markdown",
  "content": "Data-center operations · Atlas v1.3.0\n\n# Checkpointing, failure and retained useful work\n\nBalances checkpoint overhead against lost progress and recovery delay.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#checkpoint-retained-work)Job allocation across nodes, storage tiers and recovery time\n\nLearning objectiveExplain the trade between checkpoint overhead, replay and durable goodput.\n\n## The mechanism\n\nLong jobs must preserve enough state to resume after interruption. A recovery checkpoint includes the model and other required training state in a consistent version. Writing it consumes time and bandwidth; saving less often increases expected recomputation after failure. Restoring also takes time, and a local checkpoint may disappear with the node unless it has reached an adequate durability tier. Google Cloud’s goodput discussion connects checkpointing and elastic recovery to useful progress. This is distinct from activation checkpointing, which recomputes forward intermediates to save working memory during a step rather than providing durable restart protection.\n\n1.  Identify all state needed to resume the intended computation consistently.\n2.  Take a coordinated snapshot or use a supported asynchronous consistency method.\n3.  Write and verify checkpoint completion at the tier that survives the chosen failure scope.\n4.  Detect interruption, select a valid checkpoint and restore compatible resources.\n5.  Measure retained progress after replay and tune checkpoint intervals against observed interruption behavior.\n\n## Science and assumptions\n\n### Simple checkpoint loss fraction\n\n$$\nL\\approx\\frac{C}{\\tau}+\\frac{\\lambda\\tau}{2}+\\lambda R\n$$\n\nApproximate loss fraction adds checkpoint cost per interval, expected half-interval lost work per failure, and recovery time per failure.\n\nThe rare-failure approximation is not meaningful as an operating prediction when calculated overhead approaches or exceeds one.\n\nSymbols and units\n\nL\n\nApproximate fractional loss of useful work time\n\nfraction\n\nC\n\nBlocking checkpoint duration\n\ns\n\nτ\n\nInterval between checkpoints\n\ns\n\nλ\n\nSystem interruption rate in the rare-failure model\n\nfailures/s\n\nR\n\nRecovery time per interruption\n\ns\n\nC is checkpoint seconds, τ interval seconds, λ failures/s and R recovery seconds.\n\nAssumptions Rare independent failures, uniform failure timing and no overlapping checkpoint overhead; correlated failures need a richer model.\n\n## Engineering example\n\nSynthetic interval 600 s and checkpoint cost 12 s spends 2% of time saving before failures. An interruption uniformly within an interval loses about 300 s of work on average, plus restart and restore time.\n\n## Distinctions and tradeoffs\n\nAn attempted checkpoint is not a durable checkpoint, and activation rematerialization is not a recovery mechanism.\n\nFrequent checkpoints reduce expected lost progress but add write overhead and storage pressure; less frequent saves do the reverse.\n\nBusy accelerators can repeatedly redo lost work, so retained progress is a better operational measure than raw activity alone.\n\n## Specifications and interfaces\n\nRecovery cost\n\nDetection, restart, restore and replay\n\nAll are time losses at the job boundary.\n\nCheckpoint validity\n\nConsistent and sufficiently durable state\n\nA partial or local-only file may not survive the relevant failure.\n\n### Interfaces\n\nConsistent job state Input\n\nParameters, optimizer, progress and required random state\n\nRestartable checkpoint Output\n\nVersioned durable state with completion evidence\n\n## Check your understanding\n\n### What should a useful-work metric exclude after a restart?\n\n1.  All completed progress before the checkpoint\n2.  Only the final output token\n3.  Work that was lost and had to be repeated\n\nAnswer and reasoning\n\nWork that was lost and had to be repeated\n\nRepeated computation consumes resources without advancing the retained state beyond progress already achieved.\n\n## Evidence and scope\n\nFailure rates and times are synthetic. Provider-reported speedups are not imported as defaults; real failure correlation and checkpoint durability require operational evidence.\n\nCompare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests.\n\n## Sources\n\n-   [Elastic training and optimized checkpointing improve ML Goodput](https://cloud.google.com/blog/products/ai-machine-learning/elastic-training-and-optimized-checkpointing-improve-ml-goodput)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-retained-work-34f5856318637549.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n\n### Related lessons\n\n-   [Storage paths for datasets and checkpoints](https://semiaifoundry.com/learn/atlas/storage-input-checkpoint-path/)"
}