{"id":"memory-hierarchy-route","level":"compute","name":"Full memory hierarchy and input path","shortName":"Full memory hierarchy and input path","role":"Tracks data through storage, host memory, accelerator memory, caches and operand storage.","description":"A tensor does not jump directly from a dataset into an arithmetic unit. Persistent storage supplies files or objects; host memory holds decoded and prepared data; a transfer path places tensors in accelerator memory. Caches and software-managed SRAM then stage tiles close to execution, while registers hold immediate operands and accumulators. Results may return through the hierarchy or remain resident for later kernels. Each level has a different capacity, bandwidth, latency and ownership policy. The route is workload-specific: direct-storage paths can bypass some host copies, while coherent shared-memory systems alter ownership without eliminating the physical travel.","specs":[{"label":"Capacity","value":"Bytes resident at one level","context":"Allocated, usable and physically installed memory can differ."},{"label":"Bandwidth and latency","value":"Bytes/s and seconds per access","context":"High bandwidth requires enough independent work; it does not imply low individual-access latency."}],"interfaces":[{"name":"Dataset or checkpoint storage","direction":"in","signal":"Persistent bytes decoded into tensors"},{"name":"Registers and scratchpad","direction":"out","signal":"Operand tiles and partial sums consumed by arithmetic"}],"tradeoff":"Larger nearby storage improves reuse but costs area; offloading to a larger distant level increases transfer time and may expose a new bottleneck.","whyAI":"Input starvation, HBM traffic, cache misses and register spilling occur on different links and require different remedies.","related":["storage","nvme-storage","ddr-memory","register-file","memory-interface-choice"],"sources":[{"title":"CUDA C++ Programming Guide","url":"https://docs.nvidia.com/cuda/cuda-c-programming-guide/"},{"title":"What is NUMA? Linux kernel 6.6 documentation","url":"https://www.kernel.org/doc/html/v6.6/mm/numa.html"}],"scale":"Register to storage hierarchy and product lifecycle","kind":"architecture","branch":"Custom silicon ecosystem","subbranch":"Memory choices and silicon economics","mechanism":["Read persistent data and perform any required decompression, parsing and preprocessing in the supported input path.","Place prepared tensors in host or shared memory, tracking pinning, ownership and asynchronous transfer completion.","Move or map tensors into accelerator-accessible memory through the actual system interconnect.","Stage reusable tiles through caches or explicit SRAM and finally into registers or tensor operand storage.","Retain, evict or write back results according to their next consumer and lifetime, measuring traffic at each chosen boundary."],"misconception":"All memory bytes are not equally close, and a unified virtual address does not eliminate copies, migration or physical bandwidth limits.","check":{"question":"A kernel’s register use causes spills. Where should diagnosis begin?","options":["The compiler allocation and spill traffic path","Only the storage array’s capacity","The tokenizer’s vocabulary size"],"answer":0,"explanation":"Spilling moves temporary state out of registers into a less local storage path, adding traffic even when the dataset input rate is adequate."},"science":[{"title":"Boundary transfer floor","equation":"Tj ≥ Qj / Bj","explanation":"For level j, Qj is bytes crossing its boundary and Bj sustainable bytes/s. Tj is seconds of transfer demand.","assumptions":"Overlap can hide some demand, but traffic must be counted at each boundary separately; cache reuse changes Qj."}],"prerequisites":["register-file","shared-sram","ddr-memory"],"learningObjective":"Trace a tensor’s physical data path and distinguish capacity, bandwidth and latency at every level.","workstream":"W11","visualFamily":"memory-route","engineeringExample":"Synthetic route: reading a 2 GB batch from storage at 1 GB/s needs at least 2 s, even if its later HBM-to-core stage can move that amount in milliseconds. Prefetching helps only when future batches are available and buffers fit.","evidenceNotes":"The route is representative, not universal. Direct-storage and coherent-memory alternatives need their own labeled interfaces and measured traffic.","labId":"inference-placement","scope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.","investigationId":"execution","labScope":"Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied."}