The mechanism
A tensor does not jump directly from a dataset into an arithmetic unit. Persistent storage supplies files or objects; host memory holds decoded and prepared data; a transfer path places tensors in accelerator memory. Caches and software-managed SRAM then stage tiles close to execution, while registers hold immediate operands and accumulators. Results may return through the hierarchy or remain resident for later kernels. Each level has a different capacity, bandwidth, latency and ownership policy. The route is workload-specific: direct-storage paths can bypass some host copies, while coherent shared-memory systems alter ownership without eliminating the physical travel.
- Read persistent data and perform any required decompression, parsing and preprocessing in the supported input path.
- Place prepared tensors in host or shared memory, tracking pinning, ownership and asynchronous transfer completion.
- Move or map tensors into accelerator-accessible memory through the actual system interconnect.
- Stage reusable tiles through caches or explicit SRAM and finally into registers or tensor operand storage.
- Retain, evict or write back results according to their next consumer and lifetime, measuring traffic at each chosen boundary.
Science and assumptions
Boundary transfer floor
At each memory boundary, transfer demand takes at least crossing bytes divided by sustainable useful bandwidth.
Symbols and units
- Tⱼ
- Transfer time demanded at memory boundary j
s
- Qⱼ
- Payload crossing boundary j
bytes
- Bⱼ
- Sustainable useful bandwidth at that boundary
bytes/s
- j
- Memory-boundary index
integer index
For level j, Qj is bytes crossing its boundary and Bj sustainable bytes/s. Tj is seconds of transfer demand.
Assumptions Overlap can hide some demand, but traffic must be counted at each boundary separately; cache reuse changes Qj.
Engineering example
Synthetic route: reading a 2 GB batch from storage at 1 GB/s needs at least 2 s, even if its later HBM-to-core stage can move that amount in milliseconds. Prefetching helps only when future batches are available and buffers fit.
Distinctions and tradeoffs
All memory bytes are not equally close, and a unified virtual address does not eliminate copies, migration or physical bandwidth limits.
Larger nearby storage improves reuse but costs area; offloading to a larger distant level increases transfer time and may expose a new bottleneck.
Input starvation, HBM traffic, cache misses and register spilling occur on different links and require different remedies.
Specifications and interfaces
- Capacity
- Bytes resident at one level
Allocated, usable and physically installed memory can differ.
- Bandwidth and latency
- Bytes/s and seconds per access
High bandwidth requires enough independent work; it does not imply low individual-access latency.
Interfaces
- Dataset or checkpoint storage Input
- Persistent bytes decoded into tensors
- Registers and scratchpad Output
- Operand tiles and partial sums consumed by arithmetic
Check your understanding
A kernel’s register use causes spills. Where should diagnosis begin?
- The compiler allocation and spill traffic path
- Only the storage array’s capacity
- The tokenizer’s vocabulary size
Answer and reasoning
The compiler allocation and spill traffic path
Spilling moves temporary state out of registers into a less local storage path, adding traffic even when the dataset input rate is adequate.
Evidence and scope
The route is representative, not universal. Direct-storage and coherent-memory alternatives need their own labeled interfaces and measured traffic.
Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.