# When completed work survives a restart Canonical: https://semiaifoundry.com/learn/atlas/retained-work-restart-boundary/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Infrastructure · Atlas v1.3.0 # When completed work survives a restart Distinguish completed, restart-durable, unfinished and lost work under a declared failure schedule. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#retained-work-restart-boundary)One step → checkpoint → restart → observation interval Learning objectiveCalculate retained and restart-durable work from an explicit checkpoint/fault timeline, close the work and energy ledgers, and explain the limit of the result. ## The mechanism A running system can finish useful steps that have not yet reached a restart-valid checkpoint. A fault preserves the last committed checkpoint, discards later volatile progress and may interrupt a partially completed step. Recovery waits and checkpoint reads consume time and energy before new work resumes. Tracking each state prevents busy hardware, completed work and durable progress from becoming the same number. 1. Completed steps accumulate as volatile progress while the last committed checkpoint remains unchanged. 2. The example pauses execution at a whole-step boundary, writes its checkpoint and then performs an atomic commit; only that commit advances durable progress. 3. A fault preserves committed progress and transfers later completed and partial work to the lost-work ledger. 4. Repair and restore precede resumed execution. Their energy remains in the accounting, as does energy spent on work that must be repeated. 5. At the observation horizon, retained whole steps equal durable plus volatile completed steps. An unfinished step is excluded, and the horizon does not create a new checkpoint. ## Science and assumptions ### An exact work ledger for the example $$ W_{\mathrm{exec}}=D+V+F+L,\qquad W_{\mathrm{retained}}=D+V $$ Wexec = D + V + F + L; Wretained = D + V Symbols and units Wexec All executed work, including later replay whole-step equivalents D Committed restart-durable completed steps whole steps V Volatile completed steps still retained whole steps F Current unfinished step fraction, from zero up to but excluding one whole-step equivalents L Cumulative discarded completed and partial work whole-step equivalents Wretained Currently retained complete steps whole steps D is committed steps, V is volatile completed steps, F is the fraction of the current unfinished step, and L is cumulative discarded work equivalents. Every term uses the same whole-step workload definition. Assumptions One deterministic job, atomic checkpoint commits and prescribed faults. These are model state identities; useful means completed and still retained, not accuracy, convergence or application value. ### Energy per retained step $$ e=\frac{E_{\mathrm{facility}}}{W_{\mathrm{retained}}},\qquad W_{\mathrm{retained}}>0 $$ e = Efacility / Wretained, for Wretained > 0 Symbols and units e Facility energy per currently retained complete step J per step Efacility Integrated facility energy across all modeled phases J Wretained Currently retained complete steps over the same interval whole steps Facility energy integrates all active phases, including lost work, checkpoint writes, commit waits, repair and restore. Dividing by retained whole steps exposes these costs. Assumptions Fixed synthetic phase powers and constant PUE. No value is reported when zero whole steps remain. Coolant heat and facility overhead are nested energy boundaries, not extra totals to add twice. ## Engineering example Synthetic H7: run 0–8 s; write 8–9 s; commit 9–9.25 s; run 9.25–12 s; repair 12–14 s; restore 14–15 s; run 15–20 s. Retained progress is 8 + 5 = 13 whole steps; executed work is 8 + 2.75 + 5 = 15.75 equivalents. IT energy is 15.75×8 + 1.25×4 + 3×2 = 137 J; facility energy is 1.25×137 = 171.25 J. ## Distinctions and tradeoffs A completed computation is not necessarily durable after a restart, and losing its result does not erase the energy already spent. More frequent checkpointing can reduce exposed progress but adds pauses and transfer work. The preferred interval depends on the particular workload, fault schedule and checkpoint service; this lesson establishes no universal optimum. Long-running training and infrastructure workloads need recoverable state. Arithmetic activity alone does not establish that progress survives interruption. ## Specifications and interfaces Hand-checkable contract 1 s per step; one fault at 12 s Synthetic H7 fixture: checkpoint after 8 running seconds, 1 s write and 0.25 s commit; 2 s repair and 1 s restore. At the 20 s horizon 13 retained whole steps; 8 restart-durable 15.75 work equivalents executed; 2.75 discarded; no automatic checkpoint at the horizon. Energy boundary 137 J IT; 171.25 J facility Run/checkpoint/recovery powers are 8/4/2 W. Constant scenario PUE is 1.25, not an annual measured facility rating. ### Interfaces Mutable application state Input Completed and partially completed progress under one workload contract Checkpoint storage Output A complete written and committed state for restart; restore reads are distinct from writes Retained result and resource ledger Output Durable, volatile, unfinished and lost work plus energy consumed over the same observation interval ## Check your understanding ### In the H7 example, a checkpoint of step 8 commits at 9.25 s. Execution then runs at one step per second until a fault at 12 s. What survives immediately after the fault? 1. 8 durable steps survive; 2.75 later work equivalents are discarded. 2. 10 whole steps survive because all completed steps are automatically durable. 3. 10.75 useful steps survive, including the partially completed step. Answer and reasoning 8 durable steps survive; 2.75 later work equivalents are discarded. The last valid commit protects 8 steps. The 2 completed steps and 0.75 unfinished step executed afterward are volatile, so 2.75 work equivalents move to the lost-work ledger. ## Evidence and scope The source documents checkpoint completion and valid restart, not this atomic timing or its invented numerical inputs. H7 executes with the included teaching runner; the current browser hall controls do not expose every small fixture rate. No human specialist review, instructor observation or learning efficacy result is claimed. Use Engineering experiences → Rack → retained work, or the instructor pack’s H7 worked contract. ## Sources - [LLNL SCR 3.1.0 — checkpoint completion and restart](https://scr.readthedocs.io/en/latest/users/integration.html) - [DOE/LBNL — Data Center Metering and Resource Guide](https://datacenters.lbl.gov/sites/default/files/DataCenterMeteringandResourceGuide_02072017.pdf) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/retained-work-restart-boundary-a29e940810fcb313.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Checkpointing, failure and retained useful work](https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/) - [From memory placement to network, power and cooling](https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/) ### Related lessons - [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/) - [Commissioning a usable AI cluster](https://semiaifoundry.com/learn/atlas/cluster-commissioning-gates/) - [Qualify the memory system and retain useful work](https://semiaifoundry.com/learn/atlas/memory-system-qualification/)