The mechanism
Bandwidth measures a stream; latency measures the delay before a particular dependency can proceed. NAND access and the controller’s request unit can make a small useful read consume a much larger transfer. Concurrency and prefetch can hide some delays when the next address is known early, but cannot make an unexpected dependency arrive before it is requested. Sandisk’s 2025 fact sheet explicitly qualifies its performance comparison with higher HBF latency and larger pages than HBM, using a particular internal weight-reading simulation. Writes also require their own service model: NAND programming, erasure, garbage collection and wear do not disappear behind a wider host link.
- Measure the useful request size separately from controller granularity and actual fetched bytes.
- Separate first-access latency, queue time, transfer time and completion overhead.
- Issue independent reads concurrently where the software knows their addresses; budget outstanding bytes and staging space.
- Prefetch predictable state far enough ahead, measuring misses and wasted transfers when prediction fails.
- Specify write rate, program/erase behavior, amplification and endurance for the actual configuration; do not substitute a read-bandwidth grade.
Science and assumptions
Useful-byte efficiency
Buseful ≤ Bphysical u/g
Symbols and units
- Buseful
- Rate of useful bytes consumed
bytes/s
- Bphysical
- Actual transferred payload rate
bytes/s
- u
- Useful bytes from a transaction
bytes
- g
- Fetched bytes per transaction
bytes
A transaction fetches g bytes but only u are consumed before eviction. Coalescing or reuse can improve the ratio.
Assumptions 0 < u ≤ g, no later use of the remaining bytes; this is traffic efficiency, not a measured HBF result.
Concurrency to cover latency
Qflight ≥ Btarget L
Symbols and units
- Qflight
- Required outstanding payload
bytes
- Btarget
- Target transfer rate
bytes/s
- L
- Request-to-completion latency
s
Sustaining a target byte rate over latency L needs approximately that rate times latency in flight.
Assumptions Steady independent requests, adequate resources, no contention or dependency stalls; necessary capacity is not sufficient throughput proof.
Engineering example
Synthetic access: 1 TB/s with 20 µs completion latency requires at least 20 MB outstanding in the simple concurrency model. A 16 KiB fetch with only 1 KiB consumed has 6.25% useful-byte efficiency. Neither latency nor granularity is an HBF product specification.
Distinctions and tradeoffs
Useful-byte efficiency is 1/16 for this synthetic access pattern; batching nearby accesses or reuse can change it.
Aggressive prefetch can hide latency but consumes staging memory and bandwidth, especially when addresses or expert choices are unpredictable.
A model that fits in a large tier can still stall if its access pattern cannot expose enough predictable parallelism.
Specifications and interfaces
- Missing HBF defaults
- Do not invent latency or page size
Scenario inputs are hypothetical until a pinned specification or measurement supplies them.
- Reads versus writes
- Different access operations
Read-dominant does not mean physically read-only or unlimited endurance.
Interfaces
- Prefetch planner Input
- Future address, issue time and useful-byte prediction
- Staging memory Output
- Ready data plus occupancy and miss evidence
Check your understanding
A request fetches 16 KiB but consumes 1 KiB with no reuse. What fraction is immediately useful?
- 1/16
- All of it because peak bandwidth is high
- 16 times the fetched volume
Answer and reasoning
1/16
Useful-byte efficiency is 1/16 for this synthetic access pattern; batching nearby accesses or reuse can change it.
Evidence and scope
The Sandisk fact sheet’s comparison concerns internal testing/simulation of reading 8-bit pretrained Llama 3.1 405B weights, one kernel at a time on an xPU performance model. It does not establish arbitrary random reads, training writes or end-to-end serving parity. KIOXIA supports general NAND access distinctions, not HBF timing numbers.
Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.