{
  "schemaVersion": 1,
  "id": "atlas-lesson-storage-input-checkpoint-path",
  "title": "Storage paths for datasets and checkpoints",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Coordinates persistent data, local caching and durable state transfer without confusing their roles.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/storage-input-checkpoint-path/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Data-center operations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The route is representative and does not imply every platform uses the same file system or host staging. Cache and checkpoint behaviors are configuration-specific.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/storage-input-checkpoint-path-dc5f684ea90e5461.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#storage-input-checkpoint-path",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "f94bc16f3e3ef5606fc55a03b004bd219c486cfdd40b4fcdf7f66824d457e5d2",
  "contentLength": 5020,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-storage-input-checkpoint-path.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-storage-input-checkpoint-path.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Storage paths for datasets and checkpoints",
      "start": 39,
      "end": 432
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 432,
      "end": 1591
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1591,
      "end": 1619
    },
    {
      "id": "section-4",
      "title": "Input pipeline rate",
      "start": 1619,
      "end": 2449
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2449,
      "end": 2650
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2650,
      "end": 3096
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3096,
      "end": 3368
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3368,
      "end": 3563
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3563,
      "end": 3592
    },
    {
      "id": "section-10",
      "title": "Why should first-read and repeated-read timings be separated?",
      "start": 3592,
      "end": 4018
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4018,
      "end": 4350
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4350,
      "end": 4681
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4681,
      "end": 4704
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4704,
      "end": 4901
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4901,
      "end": 5020
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#storage-input-checkpoint-path"
    },
    {
      "title": "NVIDIA DGX GB Rack Scale Systems: Storage",
      "url": "https://docs.nvidia.com/dgx/dgxgb200-user-guide/storage.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/storage-input-checkpoint-path-dc5f684ea90e5461.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "NVMe solid-state storage",
      "url": "https://semiaifoundry.com/learn/atlas/nvme-storage/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    }
  ],
  "recordHash": "f1932719223b0a320aca216887b8f3e3ca36d293532a32dd498690196caa164f",
  "contentFormat": "text/markdown",
  "content": "Data-center operations · Atlas v1.3.0\n\n# Storage paths for datasets and checkpoints\n\nCoordinates persistent data, local caching and durable state transfer without confusing their roles.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#storage-input-checkpoint-path)Job allocation across nodes, storage tiers and recovery time\n\nLearning objectiveDistinguish input caching, preprocessing throughput and checkpoint durability.\n\n## The mechanism\n\nTraining input and recovery checkpoints use the same storage hierarchy for different purposes. Inputs may be read from a shared file system or object store, cached on local NVMe and transformed in host memory before reaching accelerators. Checkpoints move in the opposite direction and must satisfy a durability requirement. The DGX GB storage guide describes local caching and shared storage as separate tiers. A fast local cache can accelerate repeated reads while doing nothing for the first uncached read or for durability after node failure. Small-file metadata, decompression and bursty checkpoint writes can each limit a system that advertises high sequential bandwidth.\n\n1.  Locate the authoritative dataset and decide which repeated reads can benefit from caching.\n2.  Measure metadata, decoding and preprocessing along with byte transfer rates.\n3.  Prefetch only within a buffer budget that does not displace more valuable state.\n4.  Stage checkpoint writes and verify when they become durable for the intended failure scope.\n5.  Observe input starvation and checkpoint bursts together when sizing shared links and storage tiers.\n\n## Science and assumptions\n\n### Input pipeline rate\n\n$$\nR_{\\mathrm{useful}}\\le\\min(R_{\\mathrm{storage}},R_{\\mathrm{decode}},R_{\\mathrm{prepare}},R_{\\mathrm{transfer}})\n$$\n\nUseful input-pipeline rate cannot exceed the smallest storage, decoding, preparation or transfer rate after all use the same sample denominator.\n\nSymbols and units\n\nR\\_useful\n\nSustainable useful input-pipeline rate\n\nsamples/s\n\nR\\_storage\n\nStorage service converted to the same useful sample basis\n\nsamples/s\n\nR\\_decode\n\nDecode-stage sample rate\n\nsamples/s\n\nR\\_prepare\n\nPreparation-stage sample rate\n\nsamples/s\n\nR\\_transfer\n\nTransfer-stage service on that sample basis\n\nsamples/s\n\nEach term is converted to the same useful samples/s denominator.\n\nAssumptions Steady streaming with sufficient buffers; burstiness, shared contention and variable sample sizes require additional modeling.\n\n## Engineering example\n\nSynthetic pipeline reads 200 samples/s but decodes only 120 samples/s. Doubling raw storage bandwidth leaves the prepared-input bound at 120 samples/s until decoding improves.\n\n## Distinctions and tradeoffs\n\nA striped local cache can be fast without being durable, and a cached second read does not describe initial dataset-loading performance.\n\nLocal caching reduces repeated network traffic but duplicates data and consumes capacity; shared durability can create burst pressure when many ranks save together.\n\nAn accelerator can stall waiting for input or recovery even when its HBM and arithmetic are idle and healthy.\n\n## Specifications and interfaces\n\nInput readiness\n\nDecoded useful samples/s\n\nRaw storage bytes/s does not include parsing and preprocessing.\n\nDurability scope\n\nLocal cache versus surviving shared copy\n\nThe relevant failure domain determines whether a copy is sufficient.\n\n### Interfaces\n\nShared data and persistent state Input\n\nDataset objects, files and valid checkpoints\n\nCompute-ready batches or durable saves Output\n\nPrepared inputs and verified state transfers\n\n## Check your understanding\n\n### Why should first-read and repeated-read timings be separated?\n\n1.  The first may fetch shared data while later reads hit local cache\n2.  All reads always use the same physical path\n3.  Caching changes the number of training labels\n\nAnswer and reasoning\n\nThe first may fetch shared data while later reads hit local cache\n\nA cache changes the route and traffic, so the measurements describe different operating conditions.\n\n## Evidence and scope\n\nThe route is representative and does not imply every platform uses the same file system or host staging. Cache and checkpoint behaviors are configuration-specific.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [NVIDIA DGX GB Rack Scale Systems: Storage](https://docs.nvidia.com/dgx/dgxgb200-user-guide/storage.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/storage-input-checkpoint-path-dc5f684ea90e5461.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n-   [NVMe solid-state storage](https://semiaifoundry.com/learn/atlas/nvme-storage/)\n\n### Related lessons\n\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)"
}