{"id":"storage-input-checkpoint-path","level":"infrastructure","name":"Storage paths for datasets and checkpoints","shortName":"Storage paths for datasets and checkpoints","role":"Coordinates persistent data, local caching and durable state transfer without confusing their roles.","description":"Training input and recovery checkpoints use the same storage hierarchy for different purposes. Inputs may be read from a shared file system or object store, cached on local NVMe and transformed in host memory before reaching accelerators. Checkpoints move in the opposite direction and must satisfy a durability requirement. The DGX GB storage guide describes local caching and shared storage as separate tiers. A fast local cache can accelerate repeated reads while doing nothing for the first uncached read or for durability after node failure. Small-file metadata, decompression and bursty checkpoint writes can each limit a system that advertises high sequential bandwidth.","specs":[{"label":"Input readiness","value":"Decoded useful samples/s","context":"Raw storage bytes/s does not include parsing and preprocessing."},{"label":"Durability scope","value":"Local cache versus surviving shared copy","context":"The relevant failure domain determines whether a copy is sufficient."}],"interfaces":[{"name":"Shared data and persistent state","direction":"in","signal":"Dataset objects, files and valid checkpoints"},{"name":"Compute-ready batches or durable saves","direction":"out","signal":"Prepared inputs and verified state transfers"}],"tradeoff":"Local caching reduces repeated network traffic but duplicates data and consumes capacity; shared durability can create burst pressure when many ranks save together.","whyAI":"An accelerator can stall waiting for input or recovery even when its HBM and arithmetic are idle and healthy.","related":["nvme-storage","storage","checkpoint"],"sources":[{"title":"NVIDIA DGX GB Rack Scale Systems: Storage","url":"https://docs.nvidia.com/dgx/dgxgb200-user-guide/storage.html"}],"scale":"Job allocation across nodes, storage tiers and recovery time","kind":"architecture","branch":"Data-center operations","subbranch":"Scheduling, storage and retained work","mechanism":["Locate the authoritative dataset and decide which repeated reads can benefit from caching.","Measure metadata, decoding and preprocessing along with byte transfer rates.","Prefetch only within a buffer budget that does not displace more valuable state.","Stage checkpoint writes and verify when they become durable for the intended failure scope.","Observe input starvation and checkpoint bursts together when sizing shared links and storage tiers."],"misconception":"A striped local cache can be fast without being durable, and a cached second read does not describe initial dataset-loading performance.","check":{"question":"Why should first-read and repeated-read timings be separated?","options":["The first may fetch shared data while later reads hit local cache","All reads always use the same physical path","Caching changes the number of training labels"],"answer":0,"explanation":"A cache changes the route and traffic, so the measurements describe different operating conditions."},"science":[{"title":"Input pipeline rate","equation":"Ruseful ≤ min(Rstorage, Rdecode, Rprepare, Rtransfer)","explanation":"Each term is converted to the same useful samples/s denominator.","assumptions":"Steady streaming with sufficient buffers; burstiness, shared contention and variable sample sizes require additional modeling."}],"prerequisites":["storage","nvme-storage"],"learningObjective":"Distinguish input caching, preprocessing throughput and checkpoint durability.","workstream":"W09","visualFamily":"hall-system","engineeringExample":"Synthetic pipeline reads 200 samples/s but decodes only 120 samples/s. Doubling raw storage bandwidth leaves the prepared-input bound at 120 samples/s until decoding improves.","evidenceNotes":"The route is representative and does not imply every platform uses the same file system or host staging. Cache and checkpoint behaviors are configuration-specific.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}