# Checkpointing, failure and retained useful work

Canonical: https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Data-center operations · Atlas v1.3.0

# Checkpointing, failure and retained useful work

Balances checkpoint overhead against lost progress and recovery delay.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#checkpoint-retained-work)Job allocation across nodes, storage tiers and recovery time

Learning objectiveExplain the trade between checkpoint overhead, replay and durable goodput.

## The mechanism

Long jobs must preserve enough state to resume after interruption. A recovery checkpoint includes the model and other required training state in a consistent version. Writing it consumes time and bandwidth; saving less often increases expected recomputation after failure. Restoring also takes time, and a local checkpoint may disappear with the node unless it has reached an adequate durability tier. Google Cloud’s goodput discussion connects checkpointing and elastic recovery to useful progress. This is distinct from activation checkpointing, which recomputes forward intermediates to save working memory during a step rather than providing durable restart protection.

1.  Identify all state needed to resume the intended computation consistently.
2.  Take a coordinated snapshot or use a supported asynchronous consistency method.
3.  Write and verify checkpoint completion at the tier that survives the chosen failure scope.
4.  Detect interruption, select a valid checkpoint and restore compatible resources.
5.  Measure retained progress after replay and tune checkpoint intervals against observed interruption behavior.

## Science and assumptions

### Simple checkpoint loss fraction

$$
L\approx\frac{C}{\tau}+\frac{\lambda\tau}{2}+\lambda R
$$

Approximate loss fraction adds checkpoint cost per interval, expected half-interval lost work per failure, and recovery time per failure.

The rare-failure approximation is not meaningful as an operating prediction when calculated overhead approaches or exceeds one.

Symbols and units

L

Approximate fractional loss of useful work time

fraction

C

Blocking checkpoint duration

s

τ

Interval between checkpoints

s

λ

System interruption rate in the rare-failure model

failures/s

R

Recovery time per interruption

s

C is checkpoint seconds, τ interval seconds, λ failures/s and R recovery seconds.

Assumptions Rare independent failures, uniform failure timing and no overlapping checkpoint overhead; correlated failures need a richer model.

## Engineering example

Synthetic interval 600 s and checkpoint cost 12 s spends 2% of time saving before failures. An interruption uniformly within an interval loses about 300 s of work on average, plus restart and restore time.

## Distinctions and tradeoffs

An attempted checkpoint is not a durable checkpoint, and activation rematerialization is not a recovery mechanism.

Frequent checkpoints reduce expected lost progress but add write overhead and storage pressure; less frequent saves do the reverse.

Busy accelerators can repeatedly redo lost work, so retained progress is a better operational measure than raw activity alone.

## Specifications and interfaces

Recovery cost

Detection, restart, restore and replay

All are time losses at the job boundary.

Checkpoint validity

Consistent and sufficiently durable state

A partial or local-only file may not survive the relevant failure.

### Interfaces

Consistent job state Input

Parameters, optimizer, progress and required random state

Restartable checkpoint Output

Versioned durable state with completion evidence

## Check your understanding

### What should a useful-work metric exclude after a restart?

1.  All completed progress before the checkpoint
2.  Only the final output token
3.  Work that was lost and had to be repeated

Answer and reasoning

Work that was lost and had to be repeated

Repeated computation consumes resources without advancing the retained state beyond progress already achieved.

## Evidence and scope

Failure rates and times are synthetic. Provider-reported speedups are not imported as defaults; real failure correlation and checkpoint durability require operational evidence.

Compare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests.

## Sources

-   [Elastic training and optimized checkpointing improve ML Goodput](https://cloud.google.com/blog/products/ai-machine-learning/elastic-training-and-optimized-checkpointing-improve-ml-goodput)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-retained-work-34f5856318637549.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)
-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)

### Related lessons

-   [Storage paths for datasets and checkpoints](https://semiaifoundry.com/learn/atlas/storage-input-checkpoint-path/)