{"id":"hbm-stack","level":"package","name":"High-bandwidth memory stack","shortName":"HBM stack","role":"Stacks DRAM vertically beside the processor to provide large capacity through a very wide local interface.","description":"An HBM stack places several thin DRAM dies above a base interface and connects them vertically. It stores model weights, activations, optimizer state, and attention caches while the system is powered. The stack sits near the GPU on an advanced package, shortening the routes for a broad parallel interface. More layers generally add capacity; bandwidth also depends on interface width and per-pin transfer rate, so doubling stack height does not automatically double bandwidth. DRAM layers connect through vertical vias and package wiring to the processor, combining storage density with a wide nearby interface. Capacity, interface bandwidth and system-exposed usable memory are separate quantities because they describe different boundaries of the product.","specs":[{"label":"Micron HBM3E examples","value":"24 GB 8-high / 36 GB 12-high","context":"Vendor stack products, not a claim that both appear in GB200."},{"label":"Vendor stack bandwidth","value":"Greater than 1.2 TB/s","context":"Micron HBM3E rated capability; GPU operating point can differ."},{"label":"Interface width","value":"1,024 data bits","context":"HBM3E architecture; excludes auxiliary and power connections."}],"interfaces":[{"name":"Read/write commands","direction":"in","signal":"Requests through base interface"},{"name":"Weight and state data","direction":"out","signal":"Wide parallel transfers toward GPU"},{"name":"Power/heat","direction":"in","signal":"Electrical supply; heat leaves through package thermal paths"}],"tradeoff":"HBM offers exceptional bandwidth in a compact footprint but needs advanced stacking and packaging. Taller stacks increase integration difficulty and complicate heat removal.","whyAI":"Memory capacity determines which model states fit near the accelerator. Memory bandwidth determines how quickly those states can reach the arithmetic units.","related":["dram-die","through-silicon-via","microbump","hbm-controller","interposer","embedding","quantization"],"sources":[{"title":"Micron HBM3E","url":"https://www.micron.com/products/memory/hbm/hbm3e"},{"title":"Micron HBM3E product brief","url":"https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3Ab710d8f2-7f66-44c1-a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf"},{"title":"NVIDIA: Tesla P100 launch announcement","url":"https://nvidianews.nvidia.com/news/nvidia-delivers-massive-performance-leap-for-deep-learning-hpc-applications-with-nvidia-tesla-p100-accelerators-6622437"}],"scale":"Approximately centimeter-square stack; vendor form factor varies","kind":"exploded-hbm-stack","mechanism":["Multiple thin DRAM dies are stacked and connected vertically to form a compact memory assembly.","The stack connects to the GPU through a wide package interface that transfers many bits in parallel.","The controller repeatedly reads and writes model state while refresh preserves the volatile DRAM contents."],"misconception":"Increasing the number of stacked DRAM layers primarily adds capacity. Bandwidth also depends on interface width and operating rate, so it does not automatically scale with height.","check":{"question":"What mainly distinguishes HBM’s approach from simply raising the speed of a narrow memory bus?","options":["HBM eliminates all DRAM refresh operations.","It uses many parallel data connections close to the processor.","It stores bits permanently without electrical power."],"answer":1,"explanation":"A very wide nearby interface provides high aggregate bandwidth. Stacking adds capacity in a compact footprint, while the memory remains volatile."},"branch":"Memory hierarchy","subbranch":"Stacked DRAM","science":[{"title":"Parallel width creates aggregate bandwidth","equation":"Bpeak = Wdata × rpin / 8","explanation":"Wdata is the number of active data wires and rpin is the transfer rate in bits per second on each wire. The factor eight converts bits into bytes. Stacking more DRAM layers increases storage capacity, whereas interface width and signaling rate determine this bandwidth ceiling. These are separate design dimensions, explaining why a taller memory stack need not be a faster one.","assumptions":"Ideal interface payload rate before command, refresh, bank-conflict, or scheduling losses. Count only data wires, and use a rate definition consistent with the signaling format. Capacity and bandwidth refer to a named stack configuration."}],"history":[{"year":2016,"title":"Pascal P100 combines HBM2 packaging with NVLink","significance":"Announced on April 5, 2016, Tesla P100 brought HBM2 memory in a CoWoS package together with NVLink in NVIDIA’s Pascal accelerator platform. Faster local memory and GPU-to-GPU communication addressed data movement alongside arithmetic, anticipating the system bottlenecks of larger neural networks.","source":{"title":"NVIDIA: Tesla P100 launch announcement","url":"https://nvidianews.nvidia.com/news/nvidia-delivers-massive-performance-leap-for-deep-learning-hpc-applications-with-nvidia-tesla-p100-accelerators-6622437"}}],"learningObjective":"Separate stack height, bandwidth and delivered capacity.","engineeringExample":"Adding a memory die can increase capacity but also raises alignment, thermal and test demands; interface width is a separate choice.","evidenceNotes":"The eight-layer drawing is schematic and not a universal HBM generation.","visualFamily":"hbm-detail","prerequisites":["dram-die","through-silicon-via"],"labId":"hbm-stack-build","scope":"Adjust stack layers, thickness, joint route, interface bandwidth and conditional yield; presets are synthetic, not supplier process windows.","investigationId":"manufacturing","labScope":"Adjust stack layers, thickness, joint route, interface bandwidth and conditional yield; presets are synthetic, not supplier process windows."}