{"id":"checkpoint-retained-work","level":"infrastructure","name":"Checkpointing, failure and retained useful work","shortName":"Checkpointing, failure and retained useful work","role":"Balances checkpoint overhead against lost progress and recovery delay.","description":"Long jobs must preserve enough state to resume after interruption. A recovery checkpoint includes the model and other required training state in a consistent version. Writing it consumes time and bandwidth; saving less often increases expected recomputation after failure. Restoring also takes time, and a local checkpoint may disappear with the node unless it has reached an adequate durability tier. Google Cloud’s goodput discussion connects checkpointing and elastic recovery to useful progress. This is distinct from activation checkpointing, which recomputes forward intermediates to save working memory during a step rather than providing durable restart protection.","specs":[{"label":"Recovery cost","value":"Detection, restart, restore and replay","context":"All are time losses at the job boundary."},{"label":"Checkpoint validity","value":"Consistent and sufficiently durable state","context":"A partial or local-only file may not survive the relevant failure."}],"interfaces":[{"name":"Consistent job state","direction":"in","signal":"Parameters, optimizer, progress and required random state"},{"name":"Restartable checkpoint","direction":"out","signal":"Versioned durable state with completion evidence"}],"tradeoff":"Frequent checkpoints reduce expected lost progress but add write overhead and storage pressure; less frequent saves do the reverse.","whyAI":"Busy accelerators can repeatedly redo lost work, so retained progress is a better operational measure than raw activity alone.","related":["checkpoint","storage","storage-input-checkpoint-path"],"sources":[{"title":"Elastic training and optimized checkpointing improve ML Goodput","url":"https://cloud.google.com/blog/products/ai-machine-learning/elastic-training-and-optimized-checkpointing-improve-ml-goodput"}],"scale":"Job allocation across nodes, storage tiers and recovery time","kind":"architecture","branch":"Data-center operations","subbranch":"Scheduling, storage and retained work","mechanism":["Identify all state needed to resume the intended computation consistently.","Take a coordinated snapshot or use a supported asynchronous consistency method.","Write and verify checkpoint completion at the tier that survives the chosen failure scope.","Detect interruption, select a valid checkpoint and restore compatible resources.","Measure retained progress after replay and tune checkpoint intervals against observed interruption behavior."],"misconception":"An attempted checkpoint is not a durable checkpoint, and activation rematerialization is not a recovery mechanism.","check":{"question":"What should a useful-work metric exclude after a restart?","options":["All completed progress before the checkpoint","Only the final output token","Work that was lost and had to be repeated"],"answer":2,"explanation":"Repeated computation consumes resources without advancing the retained state beyond progress already achieved."},"science":[{"title":"Simple checkpoint loss fraction","equation":"L ≈ C/τ + λτ/2 + λR","explanation":"C is checkpoint seconds, τ interval seconds, λ failures/s and R recovery seconds.","assumptions":"Rare independent failures, uniform failure timing and no overlapping checkpoint overhead; correlated failures need a richer model."}],"prerequisites":["checkpoint","storage"],"learningObjective":"Explain the trade between checkpoint overhead, replay and durable goodput.","workstream":"W09","visualFamily":"hall-system","engineeringExample":"Synthetic interval 600 s and checkpoint cost 12 s spends 2% of time saving before failures. An interruption uniformly within an interval loses about 300 s of work on average, plus restart and restore time.","evidenceNotes":"Failure rates and times are synthetic. Provider-reported speedups are not imported as defaults; real failure correlation and checkpoint durability require operational evidence.","labId":"engineering","scope":"Compare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests.","deepLabId":"recovery","investigationId":"execution","labScope":"Compare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests."}