# Storage paths for datasets and checkpoints

Canonical: https://semiaifoundry.com/learn/atlas/storage-input-checkpoint-path/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Data-center operations · Atlas v1.3.0

# Storage paths for datasets and checkpoints

Coordinates persistent data, local caching and durable state transfer without confusing their roles.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#storage-input-checkpoint-path)Job allocation across nodes, storage tiers and recovery time

Learning objectiveDistinguish input caching, preprocessing throughput and checkpoint durability.

## The mechanism

Training input and recovery checkpoints use the same storage hierarchy for different purposes. Inputs may be read from a shared file system or object store, cached on local NVMe and transformed in host memory before reaching accelerators. Checkpoints move in the opposite direction and must satisfy a durability requirement. The DGX GB storage guide describes local caching and shared storage as separate tiers. A fast local cache can accelerate repeated reads while doing nothing for the first uncached read or for durability after node failure. Small-file metadata, decompression and bursty checkpoint writes can each limit a system that advertises high sequential bandwidth.

1.  Locate the authoritative dataset and decide which repeated reads can benefit from caching.
2.  Measure metadata, decoding and preprocessing along with byte transfer rates.
3.  Prefetch only within a buffer budget that does not displace more valuable state.
4.  Stage checkpoint writes and verify when they become durable for the intended failure scope.
5.  Observe input starvation and checkpoint bursts together when sizing shared links and storage tiers.

## Science and assumptions

### Input pipeline rate

$$
R_{\mathrm{useful}}\le\min(R_{\mathrm{storage}},R_{\mathrm{decode}},R_{\mathrm{prepare}},R_{\mathrm{transfer}})
$$

Useful input-pipeline rate cannot exceed the smallest storage, decoding, preparation or transfer rate after all use the same sample denominator.

Symbols and units

R\_useful

Sustainable useful input-pipeline rate

samples/s

R\_storage

Storage service converted to the same useful sample basis

samples/s

R\_decode

Decode-stage sample rate

samples/s

R\_prepare

Preparation-stage sample rate

samples/s

R\_transfer

Transfer-stage service on that sample basis

samples/s

Each term is converted to the same useful samples/s denominator.

Assumptions Steady streaming with sufficient buffers; burstiness, shared contention and variable sample sizes require additional modeling.

## Engineering example

Synthetic pipeline reads 200 samples/s but decodes only 120 samples/s. Doubling raw storage bandwidth leaves the prepared-input bound at 120 samples/s until decoding improves.

## Distinctions and tradeoffs

A striped local cache can be fast without being durable, and a cached second read does not describe initial dataset-loading performance.

Local caching reduces repeated network traffic but duplicates data and consumes capacity; shared durability can create burst pressure when many ranks save together.

An accelerator can stall waiting for input or recovery even when its HBM and arithmetic are idle and healthy.

## Specifications and interfaces

Input readiness

Decoded useful samples/s

Raw storage bytes/s does not include parsing and preprocessing.

Durability scope

Local cache versus surviving shared copy

The relevant failure domain determines whether a copy is sufficient.

### Interfaces

Shared data and persistent state Input

Dataset objects, files and valid checkpoints

Compute-ready batches or durable saves Output

Prepared inputs and verified state transfers

## Check your understanding

### Why should first-read and repeated-read timings be separated?

1.  The first may fetch shared data while later reads hit local cache
2.  All reads always use the same physical path
3.  Caching changes the number of training labels

Answer and reasoning

The first may fetch shared data while later reads hit local cache

A cache changes the route and traffic, so the measurements describe different operating conditions.

## Evidence and scope

The route is representative and does not imply every platform uses the same file system or host staging. Cache and checkpoint behaviors are configuration-specific.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [NVIDIA DGX GB Rack Scale Systems: Storage](https://docs.nvidia.com/dgx/dgxgb200-user-guide/storage.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/storage-input-checkpoint-path-dc5f684ea90e5461.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)
-   [NVMe solid-state storage](https://semiaifoundry.com/learn/atlas/nvme-storage/)

### Related lessons

-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)