{
  "schemaVersion": 1,
  "id": "atlas-lesson-memory-hierarchy-route",
  "title": "Full memory hierarchy and input path",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Tracks data through storage, host memory, accelerator memory, caches and operand storage.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Custom silicon ecosystem",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The route is representative, not universal. Direct-storage and coherent-memory alternatives need their own labeled interfaces and measured traffic.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-hierarchy-route-979f8294c7e13fae.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#memory-hierarchy-route",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "6fbc136f08c4467897cdc15d178791b0993771ebcd28c06c2d399b4b461bb15c",
  "contentLength": 5270,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-hierarchy-route.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-hierarchy-route.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Full memory hierarchy and input path",
      "start": 41,
      "end": 422
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 422,
      "end": 1699
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1699,
      "end": 1727
    },
    {
      "id": "section-4",
      "title": "Boundary transfer floor",
      "start": 1727,
      "end": 2344
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2344,
      "end": 2598
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2598,
      "end": 3039
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3039,
      "end": 3329
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3329,
      "end": 3509
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3509,
      "end": 3538
    },
    {
      "id": "section-10",
      "title": "A kernel’s register use causes spills. Where should diagnosis begin?",
      "start": 3538,
      "end": 3950
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3950,
      "end": 4289
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4289,
      "end": 4694
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4694,
      "end": 4717
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4717,
      "end": 4962
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4962,
      "end": 5270
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#memory-hierarchy-route"
    },
    {
      "title": "CUDA C++ Programming Guide",
      "url": "https://docs.nvidia.com/cuda/cuda-c-programming-guide/"
    },
    {
      "title": "What is NUMA? Linux kernel 6.6 documentation",
      "url": "https://www.kernel.org/doc/html/v6.6/mm/numa.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-hierarchy-route-979f8294c7e13fae.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Register file",
      "url": "https://semiaifoundry.com/learn/atlas/register-file/"
    },
    {
      "title": "Shared memory SRAM",
      "url": "https://semiaifoundry.com/learn/atlas/shared-sram/"
    },
    {
      "title": "DDR5 server memory",
      "url": "https://semiaifoundry.com/learn/atlas/ddr-memory/"
    },
    {
      "title": "Dataset and checkpoint storage hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/storage/"
    },
    {
      "title": "NVMe solid-state storage",
      "url": "https://semiaifoundry.com/learn/atlas/nvme-storage/"
    },
    {
      "title": "Choosing SRAM, HBM, DDR, LPDDR or GDDR",
      "url": "https://semiaifoundry.com/learn/atlas/memory-interface-choice/"
    }
  ],
  "recordHash": "3c6e01b3027e6456373ea76c2f5592b08c4f74d82d84ce6bf0cefa3162483bf1",
  "contentFormat": "text/markdown",
  "content": "Custom silicon ecosystem · Atlas v1.3.0\n\n# Full memory hierarchy and input path\n\nTracks data through storage, host memory, accelerator memory, caches and operand storage.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-hierarchy-route)Register to storage hierarchy and product lifecycle\n\nLearning objectiveTrace a tensor’s physical data path and distinguish capacity, bandwidth and latency at every level.\n\n## The mechanism\n\nA tensor does not jump directly from a dataset into an arithmetic unit. Persistent storage supplies files or objects; host memory holds decoded and prepared data; a transfer path places tensors in accelerator memory. Caches and software-managed SRAM then stage tiles close to execution, while registers hold immediate operands and accumulators. Results may return through the hierarchy or remain resident for later kernels. Each level has a different capacity, bandwidth, latency and ownership policy. The route is workload-specific: direct-storage paths can bypass some host copies, while coherent shared-memory systems alter ownership without eliminating the physical travel.\n\n1.  Read persistent data and perform any required decompression, parsing and preprocessing in the supported input path.\n2.  Place prepared tensors in host or shared memory, tracking pinning, ownership and asynchronous transfer completion.\n3.  Move or map tensors into accelerator-accessible memory through the actual system interconnect.\n4.  Stage reusable tiles through caches or explicit SRAM and finally into registers or tensor operand storage.\n5.  Retain, evict or write back results according to their next consumer and lifetime, measuring traffic at each chosen boundary.\n\n## Science and assumptions\n\n### Boundary transfer floor\n\n$$\nT_j\\ge\\frac{Q_j}{B_j}\n$$\n\nAt each memory boundary, transfer demand takes at least crossing bytes divided by sustainable useful bandwidth.\n\nSymbols and units\n\nTⱼ\n\nTransfer time demanded at memory boundary j\n\ns\n\nQⱼ\n\nPayload crossing boundary j\n\nbytes\n\nBⱼ\n\nSustainable useful bandwidth at that boundary\n\nbytes/s\n\nj\n\nMemory-boundary index\n\ninteger index\n\nFor level j, Qj is bytes crossing its boundary and Bj sustainable bytes/s. Tj is seconds of transfer demand.\n\nAssumptions Overlap can hide some demand, but traffic must be counted at each boundary separately; cache reuse changes Qj.\n\n## Engineering example\n\nSynthetic route: reading a 2 GB batch from storage at 1 GB/s needs at least 2 s, even if its later HBM-to-core stage can move that amount in milliseconds. Prefetching helps only when future batches are available and buffers fit.\n\n## Distinctions and tradeoffs\n\nAll memory bytes are not equally close, and a unified virtual address does not eliminate copies, migration or physical bandwidth limits.\n\nLarger nearby storage improves reuse but costs area; offloading to a larger distant level increases transfer time and may expose a new bottleneck.\n\nInput starvation, HBM traffic, cache misses and register spilling occur on different links and require different remedies.\n\n## Specifications and interfaces\n\nCapacity\n\nBytes resident at one level\n\nAllocated, usable and physically installed memory can differ.\n\nBandwidth and latency\n\nBytes/s and seconds per access\n\nHigh bandwidth requires enough independent work; it does not imply low individual-access latency.\n\n### Interfaces\n\nDataset or checkpoint storage Input\n\nPersistent bytes decoded into tensors\n\nRegisters and scratchpad Output\n\nOperand tiles and partial sums consumed by arithmetic\n\n## Check your understanding\n\n### A kernel’s register use causes spills. Where should diagnosis begin?\n\n1.  The compiler allocation and spill traffic path\n2.  Only the storage array’s capacity\n3.  The tokenizer’s vocabulary size\n\nAnswer and reasoning\n\nThe compiler allocation and spill traffic path\n\nSpilling moves temporary state out of registers into a less local storage path, adding traffic even when the dataset input rate is adequate.\n\n## Evidence and scope\n\nThe route is representative, not universal. Direct-storage and coherent-memory alternatives need their own labeled interfaces and measured traffic.\n\nPlace read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.\n\n## Sources\n\n-   [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/)\n-   [What is NUMA? Linux kernel 6.6 documentation](https://www.kernel.org/doc/html/v6.6/mm/numa.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-hierarchy-route-979f8294c7e13fae.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Register file](https://semiaifoundry.com/learn/atlas/register-file/)\n-   [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/)\n-   [DDR5 server memory](https://semiaifoundry.com/learn/atlas/ddr-memory/)\n\n### Related lessons\n\n-   [Dataset and checkpoint storage hierarchy](https://semiaifoundry.com/learn/atlas/storage/)\n-   [NVMe solid-state storage](https://semiaifoundry.com/learn/atlas/nvme-storage/)\n-   [Choosing SRAM, HBM, DDR, LPDDR or GDDR](https://semiaifoundry.com/learn/atlas/memory-interface-choice/)"
}