semiAIfoundry

System operations · Atlas v1.3.0

Distributed checkpoint and recovery state

Persists enough execution state to resume a model workflow after interruption.

Explore in AtlasDistributed software state persisted to storage

Learning objectiveConnect checkpoint size and sustained bandwidth to save time.

The mechanism

A checkpoint records a recoverable state of a training or inference workflow. Training recovery often needs more than model weights: optimizer state, step information, and other application state may be required to resume correctly. Distributed checkpoint systems can save shards from multiple workers and restore them through a coordinated load path. The atlas shows memory-state pieces leaving workers for durable storage, then returning to a new allocation. This separates a checkpoint from the transient attention KV cache and from a copy of the original dataset. Save frequency trades write overhead against potential lost work. A checkpoint is useful only when its contents, metadata, and compatibility are sufficient for the intended recovery procedure.

A checkpoint restores the kind of state it actually contains. Training checkpoints may include weights, optimizer moments and random-number state; inference state can include sequence-specific KV; a workflow journal records decisions, intents and observations. An external receipt belongs to an effect that may already have committed outside the worker. These objects should have separate identities and restore rules.

A full learning checkpoint must carry enough state to reproduce the next update. Weights-only files can reproduce a forward function while failing to reproduce training, because moments, random draws and data position differ. The lifecycle record also includes tokenizer/data/model versions, reference snapshots, stage contract and final-test exposure. Import validation checks compatibility and bounded sizes before replacing a current run. Its compact runtime fingerprint is an identifier for teaching, not an authenticated signature.

  1. Identify the state required for the intended recovery.
  2. Write its shards and metadata to durable storage.
  3. Load and coordinate those pieces when the workflow resumes.
  4. Restoration validates the saved object’s model/content or workflow identity before reusing it.
  5. A workflow restart reconciles any external keyed effect whose response was not durably recorded rather than assuming restored computation rolls the effect back.
  6. Serialize weights, moments, optimizer counter, training RNG, draw cursor and complete configuration.
  7. Validate versions, finite values, shapes and cross-field invariants before committing an import.
  8. Resume from the saved cursor and compare every subsequent state field with an uninterrupted reference.

Science and assumptions

Checkpoint write-time floor

tsave≥ScheckpointBsustainedt_{\mathrm{save}}\ge\frac{S_{\mathrm{checkpoint}}}{B_{\mathrm{sustained}}}

Checkpoint save time is at least persisted bytes divided by sustained storage bandwidth.

Symbols and units
t_save
Checkpoint write duration

s

S_checkpoint
Actual checkpoint state persisted

bytes

B_sustained
Sustained end-to-end storage payload rate

bytes/s

Checkpoint size includes the actual state being persisted, and the relevant rate is sustainable end-to-end storage throughput. Metadata and coordination add costs beyond this lower bound.

Assumptions Ideal streaming lower bound over the chosen storage path; asynchronous saves may overlap work but do not eliminate traffic or consistency requirements.

Restart the same learning experiment · worked relation

St+1=F(St,Dt),St=(θt,mt,vt,t,RNGt,cursort,config)S_{t+1}=F(S_t,D_t),\qquad S_t=(\theta_t,m_t,v_t,t,RNG_t,cursor_t,config)

Resume weights, optimizer, RNG and data cursor as one experiment state.

Symbols and units
S_t
Complete reproducible learning state

record

F
Declared deterministic update given its saved state

function

D_t
Sampled training data selected by saved RNG/cursor policy

examples

theta,m,v
Weights and optimizer moment arrays

parameters and moments

t
Optimizer counter

updates

An automated fixture trains 17 updates, exports the complete checkpoint, restores it and performs 23 more. Its weights, moments, RNG, draw counter and history exactly match 40 uninterrupted updates in the same deterministic binary64 implementation. With two sampled examples per update, the final draw count is 80. Restoring only weights while resetting moments produces a different next trajectory.

Assumptions Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract.

Engineering example

A 2 TB checkpoint at 20 GB/s has a lower-bound transfer time of 100 seconds before metadata, contention and consistency work.

The incident journal can end at a valid approved reservation intent while its environment already stores reservation-1. Restoring only that journal does not return the two consumed FILTER-A units. Recovery checks the one pending service against the environment receipt and returns the original outcome. In contrast, restoring a training checkpoint can repeat arithmetic from saved parameters but cannot by itself undo inventory in a separate service.

An automated fixture trains 17 updates, exports the complete checkpoint, restores it and performs 23 more. Its weights, moments, RNG, draw counter and history exactly match 40 uninterrupted updates in the same deterministic binary64 implementation. With two sampled examples per update, the final draw count is 80. Restoring only weights while resetting moments produces a different next trajectory.

Distinctions and tradeoffs

Saving model weights alone may not preserve the optimizer and other state needed for faithful training continuation.

Frequent saves reduce potential lost progress. They add I/O and coordination overhead.

Long training runs must tolerate interruptions. Checkpointing protects accumulated work and enables controlled continuation.

Specifications and interfaces

Training state
Weights + optimizer + application state

Exact recovery requirements depend on the workload.

Distributed storage
Sharded state

Multiple workers can save and restore coordinated pieces.

Purpose
Resume or reuse a saved state

Not the same as a live KV cache.

Interfaces

Worker state and metadata Input
Recoverable execution snapshot
Restarted workers Output
Restored compatible state

Check your understanding

Which statement about a training checkpoint is correct?

  1. It may need optimizer and application state in addition to weights
  2. It is identical to the prompt’s KV cache
  3. It guarantees infinite backup power
Answer and reasoning

It may need optimizer and application state in addition to weights

The required snapshot depends on what must be restored. Training state often extends beyond the parameter tensors.

Does restoring a training checkpoint automatically undo a committed tool reservation?

  1. Yes, all saved state is interchangeable
  2. No, the reservation is an external effect with a separate contract
  3. Only if the checkpoint is recent
Answer and reasoning

No, the reservation is an external effect with a separate contract

A training state restore does not reverse a separate external service’s committed state.

What is missing from a weights-only continuation record?

  1. Optimizer and sampling state needed to reproduce the next training update.
  2. The ability to compute any forward prediction.
  3. Only an optional chart title.
Answer and reasoning

Optimizer and sampling state needed to reproduce the next training update.

Optimizer and sampling state needed to reproduce the next training update. Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract.

Evidence and scope

The bound is an I/O boundary, not a complete recovery policy.

The local E5 journal/receipt replay is executable. It is separate from the operating hall’s retained computational-work model and from actual distributed training checkpoint formats.

Mode: executable full checkpoint. Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Compare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests.

Sources

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

Related lessons