{
  "schemaVersion": 1,
  "id": "atlas-lesson-retained-work-restart-boundary",
  "title": "When completed work survives a restart",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Distinguish completed, restart-durable, unfinished and lost work under a declared failure schedule.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/retained-work-restart-boundary/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Infrastructure",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The source documents checkpoint completion and valid restart, not this atomic timing or its invented numerical inputs. H7 executes with the included teaching runner; the current browser hall controls do not expose every small fixture rate. No human specialist review, instructor observation or learning efficacy result is claimed.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/retained-work-restart-boundary-a29e940810fcb313.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#retained-work-restart-boundary",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "fa31c877b7f459bcfaa6067f81a08030a974c10706f27783065b30e5685273aa",
  "contentLength": 7531,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-retained-work-restart-boundary.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-retained-work-restart-boundary.md",
  "sections": [
    {
      "id": "section-1",
      "title": "When completed work survives a restart",
      "start": 31,
      "end": 495
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 495,
      "end": 1637
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1637,
      "end": 1665
    },
    {
      "id": "section-4",
      "title": "An exact work ledger for the example",
      "start": 1665,
      "end": 2674
    },
    {
      "id": "section-5",
      "title": "Energy per retained step",
      "start": 2674,
      "end": 3462
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 3462,
      "end": 3784
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 3784,
      "end": 4341
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 4341,
      "end": 4866
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 4866,
      "end": 5251
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5251,
      "end": 5280
    },
    {
      "id": "section-11",
      "title": "In the H7 example, a checkpoint of step 8 commits at 9.25 s. Execution then runs at one step per second until a fault at 12 s. What survives immediately after the fault?",
      "start": 5280,
      "end": 5949
    },
    {
      "id": "section-12",
      "title": "Evidence and scope",
      "start": 5949,
      "end": 6402
    },
    {
      "id": "section-13",
      "title": "Sources",
      "start": 6402,
      "end": 6896
    },
    {
      "id": "section-14",
      "title": "Continue exploring",
      "start": 6896,
      "end": 6919
    },
    {
      "id": "section-15",
      "title": "Foundations",
      "start": 6919,
      "end": 7181
    },
    {
      "id": "section-16",
      "title": "Related lessons",
      "start": 7181,
      "end": 7531
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#retained-work-restart-boundary"
    },
    {
      "title": "LLNL SCR 3.1.0 — checkpoint completion and restart",
      "url": "https://scr.readthedocs.io/en/latest/users/integration.html"
    },
    {
      "title": "DOE/LBNL — Data Center Metering and Resource Guide",
      "url": "https://datacenters.lbl.gov/sites/default/files/DataCenterMeteringandResourceGuide_02072017.pdf"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/retained-work-restart-boundary-a29e940810fcb313.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Checkpointing, failure and retained useful work",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/"
    },
    {
      "title": "From memory placement to network, power and cooling",
      "url": "https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Commissioning a usable AI cluster",
      "url": "https://semiaifoundry.com/learn/atlas/cluster-commissioning-gates/"
    },
    {
      "title": "Qualify the memory system and retain useful work",
      "url": "https://semiaifoundry.com/learn/atlas/memory-system-qualification/"
    }
  ],
  "recordHash": "ee2364cd42803366ffcdb07c05e5190c19b9d84b58b0b65b051d15e86bce0fb2",
  "contentFormat": "text/markdown",
  "content": "Infrastructure · Atlas v1.3.0\n\n# When completed work survives a restart\n\nDistinguish completed, restart-durable, unfinished and lost work under a declared failure schedule.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#retained-work-restart-boundary)One step → checkpoint → restart → observation interval\n\nLearning objectiveCalculate retained and restart-durable work from an explicit checkpoint/fault timeline, close the work and energy ledgers, and explain the limit of the result.\n\n## The mechanism\n\nA running system can finish useful steps that have not yet reached a restart-valid checkpoint. A fault preserves the last committed checkpoint, discards later volatile progress and may interrupt a partially completed step. Recovery waits and checkpoint reads consume time and energy before new work resumes. Tracking each state prevents busy hardware, completed work and durable progress from becoming the same number.\n\n1.  Completed steps accumulate as volatile progress while the last committed checkpoint remains unchanged.\n2.  The example pauses execution at a whole-step boundary, writes its checkpoint and then performs an atomic commit; only that commit advances durable progress.\n3.  A fault preserves committed progress and transfers later completed and partial work to the lost-work ledger.\n4.  Repair and restore precede resumed execution. Their energy remains in the accounting, as does energy spent on work that must be repeated.\n5.  At the observation horizon, retained whole steps equal durable plus volatile completed steps. An unfinished step is excluded, and the horizon does not create a new checkpoint.\n\n## Science and assumptions\n\n### An exact work ledger for the example\n\n$$\nW_{\\mathrm{exec}}=D+V+F+L,\\qquad W_{\\mathrm{retained}}=D+V\n$$\n\nWexec = D + V + F + L; Wretained = D + V\n\nSymbols and units\n\nWexec\n\nAll executed work, including later replay\n\nwhole-step equivalents\n\nD\n\nCommitted restart-durable completed steps\n\nwhole steps\n\nV\n\nVolatile completed steps still retained\n\nwhole steps\n\nF\n\nCurrent unfinished step fraction, from zero up to but excluding one\n\nwhole-step equivalents\n\nL\n\nCumulative discarded completed and partial work\n\nwhole-step equivalents\n\nWretained\n\nCurrently retained complete steps\n\nwhole steps\n\nD is committed steps, V is volatile completed steps, F is the fraction of the current unfinished step, and L is cumulative discarded work equivalents. Every term uses the same whole-step workload definition.\n\nAssumptions One deterministic job, atomic checkpoint commits and prescribed faults. These are model state identities; useful means completed and still retained, not accuracy, convergence or application value.\n\n### Energy per retained step\n\n$$\ne=\\frac{E_{\\mathrm{facility}}}{W_{\\mathrm{retained}}},\\qquad W_{\\mathrm{retained}}>0\n$$\n\ne = Efacility / Wretained, for Wretained > 0\n\nSymbols and units\n\ne\n\nFacility energy per currently retained complete step\n\nJ per step\n\nEfacility\n\nIntegrated facility energy across all modeled phases\n\nJ\n\nWretained\n\nCurrently retained complete steps over the same interval\n\nwhole steps\n\nFacility energy integrates all active phases, including lost work, checkpoint writes, commit waits, repair and restore. Dividing by retained whole steps exposes these costs.\n\nAssumptions Fixed synthetic phase powers and constant PUE. No value is reported when zero whole steps remain. Coolant heat and facility overhead are nested energy boundaries, not extra totals to add twice.\n\n## Engineering example\n\nSynthetic H7: run 0–8 s; write 8–9 s; commit 9–9.25 s; run 9.25–12 s; repair 12–14 s; restore 14–15 s; run 15–20 s. Retained progress is 8 + 5 = 13 whole steps; executed work is 8 + 2.75 + 5 = 15.75 equivalents. IT energy is 15.75×8 + 1.25×4 + 3×2 = 137 J; facility energy is 1.25×137 = 171.25 J.\n\n## Distinctions and tradeoffs\n\nA completed computation is not necessarily durable after a restart, and losing its result does not erase the energy already spent.\n\nMore frequent checkpointing can reduce exposed progress but adds pauses and transfer work. The preferred interval depends on the particular workload, fault schedule and checkpoint service; this lesson establishes no universal optimum.\n\nLong-running training and infrastructure workloads need recoverable state. Arithmetic activity alone does not establish that progress survives interruption.\n\n## Specifications and interfaces\n\nHand-checkable contract\n\n1 s per step; one fault at 12 s\n\nSynthetic H7 fixture: checkpoint after 8 running seconds, 1 s write and 0.25 s commit; 2 s repair and 1 s restore.\n\nAt the 20 s horizon\n\n13 retained whole steps; 8 restart-durable\n\n15.75 work equivalents executed; 2.75 discarded; no automatic checkpoint at the horizon.\n\nEnergy boundary\n\n137 J IT; 171.25 J facility\n\nRun/checkpoint/recovery powers are 8/4/2 W. Constant scenario PUE is 1.25, not an annual measured facility rating.\n\n### Interfaces\n\nMutable application state Input\n\nCompleted and partially completed progress under one workload contract\n\nCheckpoint storage Output\n\nA complete written and committed state for restart; restore reads are distinct from writes\n\nRetained result and resource ledger Output\n\nDurable, volatile, unfinished and lost work plus energy consumed over the same observation interval\n\n## Check your understanding\n\n### In the H7 example, a checkpoint of step 8 commits at 9.25 s. Execution then runs at one step per second until a fault at 12 s. What survives immediately after the fault?\n\n1.  8 durable steps survive; 2.75 later work equivalents are discarded.\n2.  10 whole steps survive because all completed steps are automatically durable.\n3.  10.75 useful steps survive, including the partially completed step.\n\nAnswer and reasoning\n\n8 durable steps survive; 2.75 later work equivalents are discarded.\n\nThe last valid commit protects 8 steps. The 2 completed steps and 0.75 unfinished step executed afterward are volatile, so 2.75 work equivalents move to the lost-work ledger.\n\n## Evidence and scope\n\nThe source documents checkpoint completion and valid restart, not this atomic timing or its invented numerical inputs. H7 executes with the included teaching runner; the current browser hall controls do not expose every small fixture rate. No human specialist review, instructor observation or learning efficacy result is claimed.\n\nUse Engineering experiences → Rack → retained work, or the instructor pack’s H7 worked contract.\n\n## Sources\n\n-   [LLNL SCR 3.1.0 — checkpoint completion and restart](https://scr.readthedocs.io/en/latest/users/integration.html)\n-   [DOE/LBNL — Data Center Metering and Resource Guide](https://datacenters.lbl.gov/sites/default/files/DataCenterMeteringandResourceGuide_02072017.pdf)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/retained-work-restart-boundary-a29e940810fcb313.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Checkpointing, failure and retained useful work](https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/)\n-   [From memory placement to network, power and cooling](https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/)\n\n### Related lessons\n\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Commissioning a usable AI cluster](https://semiaifoundry.com/learn/atlas/cluster-commissioning-gates/)\n-   [Qualify the memory system and retain useful work](https://semiaifoundry.com/learn/atlas/memory-system-qualification/)"
}