{
  "schemaVersion": 1,
  "id": "atlas-lesson-storage",
  "title": "Dataset and checkpoint storage hierarchy",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Persists training data, model state, and results beyond accelerator working memory.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/storage/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Data pipeline",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "IOPS and sequential bandwidth have different workload meanings.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/storage-98e34d0e5e2a0a06.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#storage",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "02a384a8a9bc1f33ffc41dc5ceb26f36f6b1a638a824ee3827bf49126213d430",
  "contentLength": 4823,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-storage.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-storage.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Dataset and checkpoint storage hierarchy",
      "start": 30,
      "end": 377
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 377,
      "end": 1345
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1345,
      "end": 1373
    },
    {
      "id": "section-4",
      "title": "Bandwidth and I/O operations",
      "start": 1373,
      "end": 2104
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2104,
      "end": 2256
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2256,
      "end": 2628
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2628,
      "end": 2968
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 2968,
      "end": 3125
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3125,
      "end": 3154
    },
    {
      "id": "section-10",
      "title": "Why can a large storage system still starve GPUs?",
      "start": 3154,
      "end": 3578
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3578,
      "end": 3837
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 3837,
      "end": 4212
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4212,
      "end": 4235
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4235,
      "end": 4337
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4337,
      "end": 4823
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#storage"
    },
    {
      "title": "NVIDIA — High-Performance Storage Architecture",
      "url": "https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-b300/latest/storage-architecture.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/storage-98e34d0e5e2a0a06.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "NVMe solid-state storage",
      "url": "https://semiaifoundry.com/learn/atlas/nvme-storage/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Leaf switch and endpoint network",
      "url": "https://semiaifoundry.com/learn/atlas/leaf-switch/"
    },
    {
      "title": "Cluster scheduler and resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scheduler/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    }
  ],
  "recordHash": "179264d2d2894cf999ace5e25ab4e9947e46af12fdd381c9adb210bd9f4b81a0",
  "contentFormat": "text/markdown",
  "content": "Data pipeline · Atlas v1.3.0\n\n# Dataset and checkpoint storage hierarchy\n\nPersists training data, model state, and results beyond accelerator working memory.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#storage)Device-to-cluster storage hierarchy; capacities are workload-dependent\n\nLearning objectiveSeparate bytes read from storage from model working-set memory.\n\n## The mechanism\n\nAI storage provides the durable data behind repeated computation. Training reads samples and metadata, while checkpointing writes snapshots of model and optimizer state. These workloads stress different properties: many small operations, high sequential bandwidth, capacity, and recovery behavior may all matter. Local storage can stage data near a worker, while shared systems provide access across jobs and nodes. The atlas presents those layers separately from GPU HBM and CPU memory; storing a dataset does not mean every byte fits in accelerator memory at once. A well-supplied compute pipeline overlaps reading and preparation with execution. If storage cannot meet the workload's access pattern, expensive arithmetic hardware can wait idle.\n\n1.  Persist datasets and model-state files in suitable storage tiers.\n2.  Stage or stream the needed portions to active workers.\n3.  Write results and recoverable checkpoints back to durable storage.\n\n## Science and assumptions\n\n### Bandwidth and I/O operations\n\n$$\nB\\approx\\mathrm{IOPS}\\,S_{\\mathrm{IO}}\n$$\n\nAverage bandwidth is approximately I/O operations per second times mean bytes per operation.\n\nSymbols and units\n\nB\n\nAverage transferred byte rate\n\nbytes/s\n\nIOPS\n\nCompleted I/O rate over the selected workload\n\noperations/s\n\nS\\_IO\n\nMean transferred size per counted operation\n\nbytes/operation\n\nFor a workload with average transfer size $S_{IO}$, operations per second and transfer size together determine an approximate byte rate. Small random requests and large sequential transfers stress different limits.\n\nAssumptions Average-rate relation over a defined workload; caching, metadata operations, concurrency, and latency alter observed performance.\n\n## Engineering example\n\nIf 10 GB/s feeds a preprocessing stage for 60 seconds, 600 GB crosses that boundary; it need not reside in HBM simultaneously.\n\n## Distinctions and tradeoffs\n\nStorage capacity is not the same as storage throughput, and durable storage is not GPU working memory.\n\nCaching and staging improve locality. Capacity, consistency, and recovery requirements add complexity.\n\nTraining needs a sustained supply of examples and recoverable state. Storage performance influences useful accelerator utilization.\n\n## Specifications and interfaces\n\nDurable contents\n\nDatasets, checkpoints, outputs\n\nDifferent access patterns may require different designs.\n\nPerformance dimensions\n\nBandwidth, IOPS, latency\n\nCapacity alone does not specify performance.\n\nHierarchy\n\nLocal staging + shared persistence\n\nA conceptual architecture, not one mandatory product.\n\n### Interfaces\n\nData ingestion and checkpoints Input\n\nPersistent records and model-state snapshots\n\nWorkers and recovery Output\n\nSamples and restored state\n\n## Check your understanding\n\n### Why can a large storage system still starve GPUs?\n\n1.  Its access latency or throughput may not match the workload\n2.  Large capacity guarantees unlimited I/O\n3.  GPUs cannot read data from any software pipeline\n\nAnswer and reasoning\n\nIts access latency or throughput may not match the workload\n\nCapacity describes how much fits. Sustained delivery depends on access patterns and the performance of the full data path.\n\n## Evidence and scope\n\nIOPS and sequential bandwidth have different workload meanings.\n\nUse the manufacturing investigation to trace compatible inputs, assembly and operating constraints; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [NVIDIA — High-Performance Storage Architecture](https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-b300/latest/storage-architecture.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/storage-98e34d0e5e2a0a06.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [NVMe solid-state storage](https://semiaifoundry.com/learn/atlas/nvme-storage/)\n\n### Related lessons\n\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Leaf switch and endpoint network](https://semiaifoundry.com/learn/atlas/leaf-switch/)\n-   [Cluster scheduler and resource allocation](https://semiaifoundry.com/learn/atlas/scheduler/)\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)"
}