The mechanism
A running system can finish useful steps that have not yet reached a restart-valid checkpoint. A fault preserves the last committed checkpoint, discards later volatile progress and may interrupt a partially completed step. Recovery waits and checkpoint reads consume time and energy before new work resumes. Tracking each state prevents busy hardware, completed work and durable progress from becoming the same number.
- Completed steps accumulate as volatile progress while the last committed checkpoint remains unchanged.
- The example pauses execution at a whole-step boundary, writes its checkpoint and then performs an atomic commit; only that commit advances durable progress.
- A fault preserves committed progress and transfers later completed and partial work to the lost-work ledger.
- Repair and restore precede resumed execution. Their energy remains in the accounting, as does energy spent on work that must be repeated.
- At the observation horizon, retained whole steps equal durable plus volatile completed steps. An unfinished step is excluded, and the horizon does not create a new checkpoint.
Science and assumptions
An exact work ledger for the example
Wexec = D + V + F + L; Wretained = D + V
Symbols and units
- Wexec
- All executed work, including later replay
whole-step equivalents
- D
- Committed restart-durable completed steps
whole steps
- V
- Volatile completed steps still retained
whole steps
- F
- Current unfinished step fraction, from zero up to but excluding one
whole-step equivalents
- L
- Cumulative discarded completed and partial work
whole-step equivalents
- Wretained
- Currently retained complete steps
whole steps
D is committed steps, V is volatile completed steps, F is the fraction of the current unfinished step, and L is cumulative discarded work equivalents. Every term uses the same whole-step workload definition.
Assumptions One deterministic job, atomic checkpoint commits and prescribed faults. These are model state identities; useful means completed and still retained, not accuracy, convergence or application value.
Energy per retained step
e = Efacility / Wretained, for Wretained > 0
Symbols and units
- e
- Facility energy per currently retained complete step
J per step
- Efacility
- Integrated facility energy across all modeled phases
J
- Wretained
- Currently retained complete steps over the same interval
whole steps
Facility energy integrates all active phases, including lost work, checkpoint writes, commit waits, repair and restore. Dividing by retained whole steps exposes these costs.
Assumptions Fixed synthetic phase powers and constant PUE. No value is reported when zero whole steps remain. Coolant heat and facility overhead are nested energy boundaries, not extra totals to add twice.
Engineering example
Synthetic H7: run 0–8 s; write 8–9 s; commit 9–9.25 s; run 9.25–12 s; repair 12–14 s; restore 14–15 s; run 15–20 s. Retained progress is 8 + 5 = 13 whole steps; executed work is 8 + 2.75 + 5 = 15.75 equivalents. IT energy is 15.75×8 + 1.25×4 + 3×2 = 137 J; facility energy is 1.25×137 = 171.25 J.
Distinctions and tradeoffs
A completed computation is not necessarily durable after a restart, and losing its result does not erase the energy already spent.
More frequent checkpointing can reduce exposed progress but adds pauses and transfer work. The preferred interval depends on the particular workload, fault schedule and checkpoint service; this lesson establishes no universal optimum.
Long-running training and infrastructure workloads need recoverable state. Arithmetic activity alone does not establish that progress survives interruption.
Specifications and interfaces
- Hand-checkable contract
- 1 s per step; one fault at 12 s
Synthetic H7 fixture: checkpoint after 8 running seconds, 1 s write and 0.25 s commit; 2 s repair and 1 s restore.
- At the 20 s horizon
- 13 retained whole steps; 8 restart-durable
15.75 work equivalents executed; 2.75 discarded; no automatic checkpoint at the horizon.
- Energy boundary
- 137 J IT; 171.25 J facility
Run/checkpoint/recovery powers are 8/4/2 W. Constant scenario PUE is 1.25, not an annual measured facility rating.
Interfaces
- Mutable application state Input
- Completed and partially completed progress under one workload contract
- Checkpoint storage Output
- A complete written and committed state for restart; restore reads are distinct from writes
- Retained result and resource ledger Output
- Durable, volatile, unfinished and lost work plus energy consumed over the same observation interval
Check your understanding
In the H7 example, a checkpoint of step 8 commits at 9.25 s. Execution then runs at one step per second until a fault at 12 s. What survives immediately after the fault?
- 8 durable steps survive; 2.75 later work equivalents are discarded.
- 10 whole steps survive because all completed steps are automatically durable.
- 10.75 useful steps survive, including the partially completed step.
Answer and reasoning
8 durable steps survive; 2.75 later work equivalents are discarded.
The last valid commit protects 8 steps. The 2 completed steps and 0.75 unfinished step executed afterward are volatile, so 2.75 work equivalents move to the lost-work ledger.
Evidence and scope
The source documents checkpoint completion and valid restart, not this atomic timing or its invented numerical inputs. H7 executes with the included teaching runner; the current browser hall controls do not expose every small fixture rate. No human specialist review, instructor observation or learning efficacy result is claimed.
Use Engineering experiences → Rack → retained work, or the instructor pack’s H7 worked contract.