The mechanism
Training input and recovery checkpoints use the same storage hierarchy for different purposes. Inputs may be read from a shared file system or object store, cached on local NVMe and transformed in host memory before reaching accelerators. Checkpoints move in the opposite direction and must satisfy a durability requirement. The DGX GB storage guide describes local caching and shared storage as separate tiers. A fast local cache can accelerate repeated reads while doing nothing for the first uncached read or for durability after node failure. Small-file metadata, decompression and bursty checkpoint writes can each limit a system that advertises high sequential bandwidth.
- Locate the authoritative dataset and decide which repeated reads can benefit from caching.
- Measure metadata, decoding and preprocessing along with byte transfer rates.
- Prefetch only within a buffer budget that does not displace more valuable state.
- Stage checkpoint writes and verify when they become durable for the intended failure scope.
- Observe input starvation and checkpoint bursts together when sizing shared links and storage tiers.
Science and assumptions
Input pipeline rate
Useful input-pipeline rate cannot exceed the smallest storage, decoding, preparation or transfer rate after all use the same sample denominator.
Symbols and units
- R_useful
- Sustainable useful input-pipeline rate
samples/s
- R_storage
- Storage service converted to the same useful sample basis
samples/s
- R_decode
- Decode-stage sample rate
samples/s
- R_prepare
- Preparation-stage sample rate
samples/s
- R_transfer
- Transfer-stage service on that sample basis
samples/s
Each term is converted to the same useful samples/s denominator.
Assumptions Steady streaming with sufficient buffers; burstiness, shared contention and variable sample sizes require additional modeling.
Engineering example
Synthetic pipeline reads 200 samples/s but decodes only 120 samples/s. Doubling raw storage bandwidth leaves the prepared-input bound at 120 samples/s until decoding improves.
Distinctions and tradeoffs
A striped local cache can be fast without being durable, and a cached second read does not describe initial dataset-loading performance.
Local caching reduces repeated network traffic but duplicates data and consumes capacity; shared durability can create burst pressure when many ranks save together.
An accelerator can stall waiting for input or recovery even when its HBM and arithmetic are idle and healthy.
Specifications and interfaces
- Input readiness
- Decoded useful samples/s
Raw storage bytes/s does not include parsing and preprocessing.
- Durability scope
- Local cache versus surviving shared copy
The relevant failure domain determines whether a copy is sufficient.
Interfaces
- Shared data and persistent state Input
- Dataset objects, files and valid checkpoints
- Compute-ready batches or durable saves Output
- Prepared inputs and verified state transfers
Check your understanding
Why should first-read and repeated-read timings be separated?
- The first may fetch shared data while later reads hit local cache
- All reads always use the same physical path
- Caching changes the number of training labels
Answer and reasoning
The first may fetch shared data while later reads hit local cache
A cache changes the route and traffic, so the measurements describe different operating conditions.
Evidence and scope
The route is representative and does not imply every platform uses the same file system or host staging. Cache and checkpoint behaviors are configuration-specific.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.