{
  "schemaVersion": 1,
  "id": "atlas-lesson-hbf-latency-prefetch",
  "title": "HBF latency, transfer granularity and prefetch",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Explain why matching read bandwidth does not guarantee matching request latency or useful-byte efficiency.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/hbf-latency-prefetch/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Memory",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The Sandisk fact sheet’s comparison concerns internal testing/simulation of reading 8-bit pretrained Llama 3.1 405B weights, one kernel at a time on an xPU performance model. It does not establish arbitrary random reads, training writes or end-to-end serving parity. KIOXIA supports general NAND access distinctions, not HBF timing numbers.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/hbf-latency-prefetch-ecddf51874cb7752.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#hbf-latency-prefetch",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ce59a24a9c07c12b22c7f4c7a585f69adf0106beda7d4c67573fe1781e75c0bd",
  "contentLength": 6007,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-hbf-latency-prefetch.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-hbf-latency-prefetch.md",
  "sections": [
    {
      "id": "section-1",
      "title": "HBF latency, transfer granularity and prefetch",
      "start": 23,
      "end": 451
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 451,
      "end": 1729
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1729,
      "end": 1757
    },
    {
      "id": "section-4",
      "title": "Useful-byte efficiency",
      "start": 1757,
      "end": 2310
    },
    {
      "id": "section-5",
      "title": "Concurrency to cover latency",
      "start": 2310,
      "end": 2813
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 2813,
      "end": 3101
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 3101,
      "end": 3511
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 3511,
      "end": 3820
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 3820,
      "end": 3983
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 3983,
      "end": 4012
    },
    {
      "id": "section-11",
      "title": "A request fetches 16 KiB but consumes 1 KiB with no reuse. What fraction is immediately useful?",
      "start": 4012,
      "end": 4344
    },
    {
      "id": "section-12",
      "title": "Evidence and scope",
      "start": 4344,
      "end": 4876
    },
    {
      "id": "section-13",
      "title": "Sources",
      "start": 4876,
      "end": 5365
    },
    {
      "id": "section-14",
      "title": "Continue exploring",
      "start": 5365,
      "end": 5388
    },
    {
      "id": "section-15",
      "title": "Foundations",
      "start": 5388,
      "end": 5633
    },
    {
      "id": "section-16",
      "title": "Related lessons",
      "start": 5633,
      "end": 6007
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#hbf-latency-prefetch"
    },
    {
      "title": "Sandisk — HBF fact sheet",
      "url": "https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf"
    },
    {
      "title": "KIOXIA — NAND erase operations and latency",
      "url": "https://www.kioxia.com/en-jp/rd/technology/topics/topics-51.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/hbf-latency-prefetch-ecddf51874cb7752.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "HBF: a read-oriented tier and its evidence boundary",
      "url": "https://semiaifoundry.com/learn/atlas/hbf-read-tier-contract/"
    },
    {
      "title": "Full memory hierarchy and input path",
      "url": "https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/"
    },
    {
      "title": "MoE experts: resident capacity, routing and misses",
      "url": "https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/"
    },
    {
      "title": "Prefill, decode and the moving KV working set",
      "url": "https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/"
    },
    {
      "title": "Kernel tiling, fusion and operand reuse",
      "url": "https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/"
    }
  ],
  "recordHash": "96897ed269c8f8e99949581b91c2de5128045eb0d3dd43fbcd32e369b76966d8",
  "contentFormat": "text/markdown",
  "content": "Memory · Atlas v1.3.0\n\n# HBF latency, transfer granularity and prefetch\n\nExplain why matching read bandwidth does not guarantee matching request latency or useful-byte efficiency.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#hbf-latency-prefetch)Request granularity, queues and staging buffers; abstract dataflow\n\nLearning objectiveExplain why matching read bandwidth does not guarantee matching request latency or useful-byte efficiency.\n\n## The mechanism\n\nBandwidth measures a stream; latency measures the delay before a particular dependency can proceed. NAND access and the controller’s request unit can make a small useful read consume a much larger transfer. Concurrency and prefetch can hide some delays when the next address is known early, but cannot make an unexpected dependency arrive before it is requested. Sandisk’s 2025 fact sheet explicitly qualifies its performance comparison with higher HBF latency and larger pages than HBM, using a particular internal weight-reading simulation. Writes also require their own service model: NAND programming, erasure, garbage collection and wear do not disappear behind a wider host link.\n\n1.  Measure the useful request size separately from controller granularity and actual fetched bytes.\n2.  Separate first-access latency, queue time, transfer time and completion overhead.\n3.  Issue independent reads concurrently where the software knows their addresses; budget outstanding bytes and staging space.\n4.  Prefetch predictable state far enough ahead, measuring misses and wasted transfers when prediction fails.\n5.  Specify write rate, program/erase behavior, amplification and endurance for the actual configuration; do not substitute a read-bandwidth grade.\n\n## Science and assumptions\n\n### Useful-byte efficiency\n\n$$\nB_{\\mathrm{useful}}\\leq B_{\\mathrm{physical}}\\frac{u}{g}\n$$\n\nBuseful ≤ Bphysical u/g\n\nSymbols and units\n\nBuseful\n\nRate of useful bytes consumed\n\nbytes/s\n\nBphysical\n\nActual transferred payload rate\n\nbytes/s\n\nu\n\nUseful bytes from a transaction\n\nbytes\n\ng\n\nFetched bytes per transaction\n\nbytes\n\nA transaction fetches g bytes but only u are consumed before eviction. Coalescing or reuse can improve the ratio.\n\nAssumptions 0 < u ≤ g, no later use of the remaining bytes; this is traffic efficiency, not a measured HBF result.\n\n### Concurrency to cover latency\n\n$$\nQ_{\\mathrm{flight}}\\geq B_{\\mathrm{target}}L\n$$\n\nQflight ≥ Btarget L\n\nSymbols and units\n\nQflight\n\nRequired outstanding payload\n\nbytes\n\nBtarget\n\nTarget transfer rate\n\nbytes/s\n\nL\n\nRequest-to-completion latency\n\ns\n\nSustaining a target byte rate over latency L needs approximately that rate times latency in flight.\n\nAssumptions Steady independent requests, adequate resources, no contention or dependency stalls; necessary capacity is not sufficient throughput proof.\n\n## Engineering example\n\nSynthetic access: 1 TB/s with 20 µs completion latency requires at least 20 MB outstanding in the simple concurrency model. A 16 KiB fetch with only 1 KiB consumed has 6.25% useful-byte efficiency. Neither latency nor granularity is an HBF product specification.\n\n## Distinctions and tradeoffs\n\nUseful-byte efficiency is 1/16 for this synthetic access pattern; batching nearby accesses or reuse can change it.\n\nAggressive prefetch can hide latency but consumes staging memory and bandwidth, especially when addresses or expert choices are unpredictable.\n\nA model that fits in a large tier can still stall if its access pattern cannot expose enough predictable parallelism.\n\n## Specifications and interfaces\n\nMissing HBF defaults\n\nDo not invent latency or page size\n\nScenario inputs are hypothetical until a pinned specification or measurement supplies them.\n\nReads versus writes\n\nDifferent access operations\n\nRead-dominant does not mean physically read-only or unlimited endurance.\n\n### Interfaces\n\nPrefetch planner Input\n\nFuture address, issue time and useful-byte prediction\n\nStaging memory Output\n\nReady data plus occupancy and miss evidence\n\n## Check your understanding\n\n### A request fetches 16 KiB but consumes 1 KiB with no reuse. What fraction is immediately useful?\n\n1.  1/16\n2.  All of it because peak bandwidth is high\n3.  16 times the fetched volume\n\nAnswer and reasoning\n\n1/16\n\nUseful-byte efficiency is 1/16 for this synthetic access pattern; batching nearby accesses or reuse can change it.\n\n## Evidence and scope\n\nThe Sandisk fact sheet’s comparison concerns internal testing/simulation of reading 8-bit pretrained Llama 3.1 405B weights, one kernel at a time on an xPU performance model. It does not establish arbitrary random reads, training writes or end-to-end serving parity. KIOXIA supports general NAND access distinctions, not HBF timing numbers.\n\nPlace read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.\n\n## Sources\n\n-   [Sandisk — HBF fact sheet](https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf)\n-   [KIOXIA — NAND erase operations and latency](https://www.kioxia.com/en-jp/rd/technology/topics/topics-51.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/hbf-latency-prefetch-ecddf51874cb7752.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [HBF: a read-oriented tier and its evidence boundary](https://semiaifoundry.com/learn/atlas/hbf-read-tier-contract/)\n-   [Full memory hierarchy and input path](https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/)\n\n### Related lessons\n\n-   [MoE experts: resident capacity, routing and misses](https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/)\n-   [Prefill, decode and the moving KV working set](https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/)\n-   [Kernel tiling, fusion and operand reuse](https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/)"
}