{
  "schemaVersion": 1,
  "id": "atlas-lesson-checkpoint",
  "title": "Distributed checkpoint and recovery state",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Persists enough execution state to resume a model workflow after interruption.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/checkpoint/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "System operations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The bound is an I/O boundary, not a complete recovery policy.\n\nThe local E5 journal/receipt replay is executable. It is separate from the operating hall’s retained computational-work model and from actual distributed training checkpoint formats.\n\nMode: executable full checkpoint. Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-71d8146fe35d9574.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#checkpoint",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ead48a54e3ef4b971238a9058b72da897c9c0dae37c17afddf3ddc9e991a9684",
  "contentLength": 10523,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-checkpoint.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Distributed checkpoint and recovery state",
      "start": 34,
      "end": 355
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 355,
      "end": 2848
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2848,
      "end": 2876
    },
    {
      "id": "section-4",
      "title": "Checkpoint write-time floor",
      "start": 2876,
      "end": 3616
    },
    {
      "id": "section-5",
      "title": "Restart the same learning experiment · worked relation",
      "start": 3616,
      "end": 4739
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4739,
      "end": 5743
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5743,
      "end": 6106
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 6106,
      "end": 6419
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 6419,
      "end": 6553
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 6553,
      "end": 6582
    },
    {
      "id": "section-11",
      "title": "Which statement about a training checkpoint is correct?",
      "start": 6582,
      "end": 7006
    },
    {
      "id": "section-12",
      "title": "Does restoring a training checkpoint automatically undo a committed tool reservation?",
      "start": 7006,
      "end": 7430
    },
    {
      "id": "section-13",
      "title": "What is missing from a weights-only continuation record?",
      "start": 7430,
      "end": 8012
    },
    {
      "id": "section-14",
      "title": "Evidence and scope",
      "start": 8012,
      "end": 8848
    },
    {
      "id": "section-15",
      "title": "Sources",
      "start": 8848,
      "end": 9357
    },
    {
      "id": "section-16",
      "title": "Continue exploring",
      "start": 9357,
      "end": 9380
    },
    {
      "id": "section-17",
      "title": "Foundations",
      "start": 9380,
      "end": 9493
    },
    {
      "id": "section-18",
      "title": "Related lessons",
      "start": 9493,
      "end": 10523
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#checkpoint"
    },
    {
      "title": "PyTorch — Distributed Checkpoint",
      "url": "https://docs.pytorch.org/docs/stable/distributed.checkpoint.html"
    },
    {
      "title": "Temporal Workflow Execution",
      "url": "https://docs.temporal.io/workflow-execution"
    },
    {
      "title": "Temporal Activities",
      "url": "https://docs.temporal.io/activities"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-71d8146fe35d9574.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "Cluster scheduler and resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scheduler/"
    },
    {
      "title": "Data, tensor, pipeline, and state parallelism",
      "url": "https://semiaifoundry.com/learn/atlas/parallelism/"
    },
    {
      "title": "Uninterruptible power supply and backup chain",
      "url": "https://semiaifoundry.com/learn/atlas/ups/"
    },
    {
      "title": "Durable actions and the ambiguous timeout",
      "url": "https://semiaifoundry.com/learn/atlas/durable-actions-idempotency/"
    },
    {
      "title": "Context, retrieval and durable memory",
      "url": "https://semiaifoundry.com/learn/atlas/context-memory-provenance/"
    },
    {
      "title": "When completed work survives a restart",
      "url": "https://semiaifoundry.com/learn/atlas/retained-work-restart-boundary/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Initialization, learning rate and stable updates",
      "url": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/"
    },
    {
      "title": "Gradient accumulation under a fixed global objective",
      "url": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/"
    }
  ],
  "recordHash": "96d6d682650e8f957b0316d81c93263c3ae093a65b042908ae8eae68ad111f19",
  "contentFormat": "text/markdown",
  "content": "System operations · Atlas v1.3.0\n\n# Distributed checkpoint and recovery state\n\nPersists enough execution state to resume a model workflow after interruption.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#checkpoint)Distributed software state persisted to storage\n\nLearning objectiveConnect checkpoint size and sustained bandwidth to save time.\n\n## The mechanism\n\nA checkpoint records a recoverable state of a training or inference workflow. Training recovery often needs more than model weights: optimizer state, step information, and other application state may be required to resume correctly. Distributed checkpoint systems can save shards from multiple workers and restore them through a coordinated load path. The atlas shows memory-state pieces leaving workers for durable storage, then returning to a new allocation. This separates a checkpoint from the transient attention KV cache and from a copy of the original dataset. Save frequency trades write overhead against potential lost work. A checkpoint is useful only when its contents, metadata, and compatibility are sufficient for the intended recovery procedure.\n\nA checkpoint restores the kind of state it actually contains. Training checkpoints may include weights, optimizer moments and random-number state; inference state can include sequence-specific KV; a workflow journal records decisions, intents and observations. An external receipt belongs to an effect that may already have committed outside the worker. These objects should have separate identities and restore rules.\n\nA full learning checkpoint must carry enough state to reproduce the next update. Weights-only files can reproduce a forward function while failing to reproduce training, because moments, random draws and data position differ. The lifecycle record also includes tokenizer/data/model versions, reference snapshots, stage contract and final-test exposure. Import validation checks compatibility and bounded sizes before replacing a current run. Its compact runtime fingerprint is an identifier for teaching, not an authenticated signature.\n\n1.  Identify the state required for the intended recovery.\n2.  Write its shards and metadata to durable storage.\n3.  Load and coordinate those pieces when the workflow resumes.\n4.  Restoration validates the saved object’s model/content or workflow identity before reusing it.\n5.  A workflow restart reconciles any external keyed effect whose response was not durably recorded rather than assuming restored computation rolls the effect back.\n6.  Serialize weights, moments, optimizer counter, training RNG, draw cursor and complete configuration.\n7.  Validate versions, finite values, shapes and cross-field invariants before committing an import.\n8.  Resume from the saved cursor and compare every subsequent state field with an uninterrupted reference.\n\n## Science and assumptions\n\n### Checkpoint write-time floor\n\n$$\nt_{\\mathrm{save}}\\ge\\frac{S_{\\mathrm{checkpoint}}}{B_{\\mathrm{sustained}}}\n$$\n\nCheckpoint save time is at least persisted bytes divided by sustained storage bandwidth.\n\nSymbols and units\n\nt\\_save\n\nCheckpoint write duration\n\ns\n\nS\\_checkpoint\n\nActual checkpoint state persisted\n\nbytes\n\nB\\_sustained\n\nSustained end-to-end storage payload rate\n\nbytes/s\n\nCheckpoint size includes the actual state being persisted, and the relevant rate is sustainable end-to-end storage throughput. Metadata and coordination add costs beyond this lower bound.\n\nAssumptions Ideal streaming lower bound over the chosen storage path; asynchronous saves may overlap work but do not eliminate traffic or consistency requirements.\n\n### Restart the same learning experiment · worked relation\n\n$$\nS_{t+1}=F(S_t,D_t),\\qquad S_t=(\\theta_t,m_t,v_t,t,RNG_t,cursor_t,config)\n$$\n\nResume weights, optimizer, RNG and data cursor as one experiment state.\n\nSymbols and units\n\nS\\_t\n\nComplete reproducible learning state\n\nrecord\n\nF\n\nDeclared deterministic update given its saved state\n\nfunction\n\nD\\_t\n\nSampled training data selected by saved RNG/cursor policy\n\nexamples\n\ntheta,m,v\n\nWeights and optimizer moment arrays\n\nparameters and moments\n\nt\n\nOptimizer counter\n\nupdates\n\nAn automated fixture trains 17 updates, exports the complete checkpoint, restores it and performs 23 more. Its weights, moments, RNG, draw counter and history exactly match 40 uninterrupted updates in the same deterministic binary64 implementation. With two sampled examples per update, the final draw count is 80. Restoring only weights while resetting moments produces a different next trajectory.\n\nAssumptions Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract.\n\n## Engineering example\n\nA 2 TB checkpoint at 20 GB/s has a lower-bound transfer time of 100 seconds before metadata, contention and consistency work.\n\nThe incident journal can end at a valid approved reservation intent while its environment already stores reservation-1. Restoring only that journal does not return the two consumed FILTER-A units. Recovery checks the one pending service against the environment receipt and returns the original outcome. In contrast, restoring a training checkpoint can repeat arithmetic from saved parameters but cannot by itself undo inventory in a separate service.\n\nAn automated fixture trains 17 updates, exports the complete checkpoint, restores it and performs 23 more. Its weights, moments, RNG, draw counter and history exactly match 40 uninterrupted updates in the same deterministic binary64 implementation. With two sampled examples per update, the final draw count is 80. Restoring only weights while resetting moments produces a different next trajectory.\n\n## Distinctions and tradeoffs\n\nSaving model weights alone may not preserve the optimizer and other state needed for faithful training continuation.\n\nFrequent saves reduce potential lost progress. They add I/O and coordination overhead.\n\nLong training runs must tolerate interruptions. Checkpointing protects accumulated work and enables controlled continuation.\n\n## Specifications and interfaces\n\nTraining state\n\nWeights + optimizer + application state\n\nExact recovery requirements depend on the workload.\n\nDistributed storage\n\nSharded state\n\nMultiple workers can save and restore coordinated pieces.\n\nPurpose\n\nResume or reuse a saved state\n\nNot the same as a live KV cache.\n\n### Interfaces\n\nWorker state and metadata Input\n\nRecoverable execution snapshot\n\nRestarted workers Output\n\nRestored compatible state\n\n## Check your understanding\n\n### Which statement about a training checkpoint is correct?\n\n1.  It may need optimizer and application state in addition to weights\n2.  It is identical to the prompt’s KV cache\n3.  It guarantees infinite backup power\n\nAnswer and reasoning\n\nIt may need optimizer and application state in addition to weights\n\nThe required snapshot depends on what must be restored. Training state often extends beyond the parameter tensors.\n\n### Does restoring a training checkpoint automatically undo a committed tool reservation?\n\n1.  Yes, all saved state is interchangeable\n2.  No, the reservation is an external effect with a separate contract\n3.  Only if the checkpoint is recent\n\nAnswer and reasoning\n\nNo, the reservation is an external effect with a separate contract\n\nA training state restore does not reverse a separate external service’s committed state.\n\n### What is missing from a weights-only continuation record?\n\n1.  Optimizer and sampling state needed to reproduce the next training update.\n2.  The ability to compute any forward prediction.\n3.  Only an optional chart title.\n\nAnswer and reasoning\n\nOptimizer and sampling state needed to reproduce the next training update.\n\nOptimizer and sampling state needed to reproduce the next training update. Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract.\n\n## Evidence and scope\n\nThe bound is an I/O boundary, not a complete recovery policy.\n\nThe local E5 journal/receipt replay is executable. It is separate from the operating hall’s retained computational-work model and from actual distributed training checkpoint formats.\n\nMode: executable full checkpoint. Exact equality is tested within one numerical implementation and operation order. Cross-device floating-point kernels can require a tolerance or different reproducibility contract. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nCompare checkpoint overhead, interruptions and retained progress; storage transfer behavior and full system qualification remain separate tests.\n\n## Sources\n\n-   [PyTorch — Distributed Checkpoint](https://docs.pytorch.org/docs/stable/distributed.checkpoint.html)\n-   [Temporal Workflow Execution](https://docs.temporal.io/workflow-execution)\n-   [Temporal Activities](https://docs.temporal.io/activities)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/checkpoint-71d8146fe35d9574.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n\n### Related lessons\n\n-   [Cluster scheduler and resource allocation](https://semiaifoundry.com/learn/atlas/scheduler/)\n-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)\n-   [Uninterruptible power supply and backup chain](https://semiaifoundry.com/learn/atlas/ups/)\n-   [Durable actions and the ambiguous timeout](https://semiaifoundry.com/learn/atlas/durable-actions-idempotency/)\n-   [Context, retrieval and durable memory](https://semiaifoundry.com/learn/atlas/context-memory-provenance/)\n-   [When completed work survives a restart](https://semiaifoundry.com/learn/atlas/retained-work-restart-boundary/)\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)\n-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)"
}