{
  "schemaVersion": 1,
  "id": "atlas-lesson-hbm-stack",
  "title": "High-bandwidth memory stack",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Stacks DRAM vertically beside the processor to provide large capacity through a very wide local interface.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/hbm-stack/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Memory hierarchy",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The eight-layer drawing is schematic and not a universal HBM generation.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/hbm-stack-d15a25e66bf44c7a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#hbm-stack",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "c79d9d011be15ad4b5876513e4b2af781ab5c4c16f080abfa5ebb6f45f9e98b4",
  "contentLength": 6533,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-hbm-stack.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-hbm-stack.md",
  "sections": [
    {
      "id": "section-1",
      "title": "High-bandwidth memory stack",
      "start": 33,
      "end": 379
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 379,
      "end": 1474
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1474,
      "end": 1502
    },
    {
      "id": "section-4",
      "title": "Parallel width creates aggregate bandwidth",
      "start": 1502,
      "end": 2584
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2584,
      "end": 2742
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2742,
      "end": 3287
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3287,
      "end": 3645
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3645,
      "end": 3867
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3867,
      "end": 3896
    },
    {
      "id": "section-10",
      "title": "What mainly distinguishes HBM’s approach from simply raising the speed of a narrow memory bus?",
      "start": 3896,
      "end": 4402
    },
    {
      "id": "section-11",
      "title": "Historical context",
      "start": 4402,
      "end": 4425
    },
    {
      "id": "section-12",
      "title": "2016 · Pascal P100 combines HBM2 packaging with NVLink",
      "start": 4425,
      "end": 4982
    },
    {
      "id": "section-13",
      "title": "Evidence and scope",
      "start": 4982,
      "end": 5221
    },
    {
      "id": "section-14",
      "title": "Sources",
      "start": 5221,
      "end": 5868
    },
    {
      "id": "section-15",
      "title": "Continue exploring",
      "start": 5868,
      "end": 5891
    },
    {
      "id": "section-16",
      "title": "Foundations",
      "start": 5891,
      "end": 6076
    },
    {
      "id": "section-17",
      "title": "Related lessons",
      "start": 6076,
      "end": 6533
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#hbm-stack"
    },
    {
      "title": "NVIDIA: Tesla P100 launch announcement",
      "url": "https://nvidianews.nvidia.com/news/nvidia-delivers-massive-performance-leap-for-deep-learning-hpc-applications-with-nvidia-tesla-p100-accelerators-6622437"
    },
    {
      "title": "Micron HBM3E",
      "url": "https://www.micron.com/products/memory/hbm/hbm3e"
    },
    {
      "title": "Micron HBM3E product brief",
      "url": "https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3Ab710d8f2-7f66-44c1-a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/hbm-stack-d15a25e66bf44c7a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "DRAM die and memory banks",
      "url": "https://semiaifoundry.com/learn/atlas/dram-die/"
    },
    {
      "title": "Through-silicon via",
      "url": "https://semiaifoundry.com/learn/atlas/through-silicon-via/"
    },
    {
      "title": "Microbumps and bonding joints",
      "url": "https://semiaifoundry.com/learn/atlas/microbump/"
    },
    {
      "title": "HBM controller and PHY",
      "url": "https://semiaifoundry.com/learn/atlas/hbm-controller/"
    },
    {
      "title": "Interposer and redistribution layers",
      "url": "https://semiaifoundry.com/learn/atlas/interposer/"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Quantization and numerical representation",
      "url": "https://semiaifoundry.com/learn/atlas/quantization/"
    }
  ],
  "recordHash": "7516a88d3157496424e84f73dfdd7896ef04537be6311041cc3978a3a8263ee7",
  "contentFormat": "text/markdown",
  "content": "Memory hierarchy · Atlas v1.3.0\n\n# High-bandwidth memory stack\n\nStacks DRAM vertically beside the processor to provide large capacity through a very wide local interface.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#hbm-stack)Approximately centimeter-square stack; vendor form factor varies\n\nLearning objectiveSeparate stack height, bandwidth and delivered capacity.\n\n## The mechanism\n\nAn HBM stack places several thin DRAM dies above a base interface and connects them vertically. It stores model weights, activations, optimizer state, and attention caches while the system is powered. The stack sits near the GPU on an advanced package, shortening the routes for a broad parallel interface. More layers generally add capacity; bandwidth also depends on interface width and per-pin transfer rate, so doubling stack height does not automatically double bandwidth. DRAM layers connect through vertical vias and package wiring to the processor, combining storage density with a wide nearby interface. Capacity, interface bandwidth and system-exposed usable memory are separate quantities because they describe different boundaries of the product.\n\n1.  Multiple thin DRAM dies are stacked and connected vertically to form a compact memory assembly.\n2.  The stack connects to the GPU through a wide package interface that transfers many bits in parallel.\n3.  The controller repeatedly reads and writes model state while refresh preserves the volatile DRAM contents.\n\n## Science and assumptions\n\n### Parallel width creates aggregate bandwidth\n\n$$\nB_{\\mathrm{peak}}=\\frac{W_{\\mathrm{data}}r_{\\mathrm{pin}}}{8}\n$$\n\nPeak bandwidth equals active data-wire count times effective bit rate per wire divided by eight bits per byte.\n\nSymbols and units\n\nB\\_peak\n\nPeak interface byte rate before losses\n\nbytes/s\n\nW\\_data\n\nNumber of active parallel data wires\n\nwires\n\nr\\_pin\n\nEffective bit rate on each data wire\n\nbits/(s·wire)\n\n8\n\nBits per byte\n\nbits/byte\n\nWdata is the number of active data wires and rpin is the transfer rate in bits per second on each wire. The factor eight converts bits into bytes. Stacking more DRAM layers increases storage capacity, whereas interface width and signaling rate determine this bandwidth ceiling. These are separate design dimensions, explaining why a taller memory stack need not be a faster one.\n\nAssumptions Ideal interface payload rate before command, refresh, bank-conflict, or scheduling losses. Count only data wires, and use a rate definition consistent with the signaling format. Capacity and bandwidth refer to a named stack configuration.\n\n## Engineering example\n\nAdding a memory die can increase capacity but also raises alignment, thermal and test demands; interface width is a separate choice.\n\n## Distinctions and tradeoffs\n\nIncreasing the number of stacked DRAM layers primarily adds capacity. Bandwidth also depends on interface width and operating rate, so it does not automatically scale with height.\n\nHBM offers exceptional bandwidth in a compact footprint but needs advanced stacking and packaging. Taller stacks increase integration difficulty and complicate heat removal.\n\nMemory capacity determines which model states fit near the accelerator. Memory bandwidth determines how quickly those states can reach the arithmetic units.\n\n## Specifications and interfaces\n\nMicron HBM3E examples\n\n24 GB 8-high / 36 GB 12-high\n\nVendor stack products, not a claim that both appear in GB200.\n\nVendor stack bandwidth\n\nGreater than 1.2 TB/s\n\nMicron HBM3E rated capability; GPU operating point can differ.\n\nInterface width\n\n1,024 data bits\n\nHBM3E architecture; excludes auxiliary and power connections.\n\n### Interfaces\n\nRead/write commands Input\n\nRequests through base interface\n\nWeight and state data Output\n\nWide parallel transfers toward GPU\n\nPower/heat Input\n\nElectrical supply; heat leaves through package thermal paths\n\n## Check your understanding\n\n### What mainly distinguishes HBM’s approach from simply raising the speed of a narrow memory bus?\n\n1.  HBM eliminates all DRAM refresh operations.\n2.  It uses many parallel data connections close to the processor.\n3.  It stores bits permanently without electrical power.\n\nAnswer and reasoning\n\nIt uses many parallel data connections close to the processor.\n\nA very wide nearby interface provides high aggregate bandwidth. Stacking adds capacity in a compact footprint, while the memory remains volatile.\n\n## Historical context\n\n### 2016 · Pascal P100 combines HBM2 packaging with NVLink\n\nAnnounced on April 5, 2016, Tesla P100 brought HBM2 memory in a CoWoS package together with NVLink in NVIDIA’s Pascal accelerator platform. Faster local memory and GPU-to-GPU communication addressed data movement alongside arithmetic, anticipating the system bottlenecks of larger neural networks.\n\n[NVIDIA: Tesla P100 launch announcement](https://nvidianews.nvidia.com/news/nvidia-delivers-massive-performance-leap-for-deep-learning-hpc-applications-with-nvidia-tesla-p100-accelerators-6622437)\n\n## Evidence and scope\n\nThe eight-layer drawing is schematic and not a universal HBM generation.\n\nAdjust stack layers, thickness, joint route, interface bandwidth and conditional yield; presets are synthetic, not supplier process windows.\n\n## Sources\n\n-   [Micron HBM3E](https://www.micron.com/products/memory/hbm/hbm3e)\n-   [Micron HBM3E product brief](https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3Ab710d8f2-7f66-44c1-a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf)\n-   [NVIDIA: Tesla P100 launch announcement](https://nvidianews.nvidia.com/news/nvidia-delivers-massive-performance-leap-for-deep-learning-hpc-applications-with-nvidia-tesla-p100-accelerators-6622437)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/hbm-stack-d15a25e66bf44c7a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [DRAM die and memory banks](https://semiaifoundry.com/learn/atlas/dram-die/)\n-   [Through-silicon via](https://semiaifoundry.com/learn/atlas/through-silicon-via/)\n\n### Related lessons\n\n-   [Microbumps and bonding joints](https://semiaifoundry.com/learn/atlas/microbump/)\n-   [HBM controller and PHY](https://semiaifoundry.com/learn/atlas/hbm-controller/)\n-   [Interposer and redistribution layers](https://semiaifoundry.com/learn/atlas/interposer/)\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Quantization and numerical representation](https://semiaifoundry.com/learn/atlas/quantization/)"
}