The mechanism
An HBM stack places several thin DRAM dies above a base interface and connects them vertically. It stores model weights, activations, optimizer state, and attention caches while the system is powered. The stack sits near the GPU on an advanced package, shortening the routes for a broad parallel interface. More layers generally add capacity; bandwidth also depends on interface width and per-pin transfer rate, so doubling stack height does not automatically double bandwidth. DRAM layers connect through vertical vias and package wiring to the processor, combining storage density with a wide nearby interface. Capacity, interface bandwidth and system-exposed usable memory are separate quantities because they describe different boundaries of the product.
- Multiple thin DRAM dies are stacked and connected vertically to form a compact memory assembly.
- The stack connects to the GPU through a wide package interface that transfers many bits in parallel.
- The controller repeatedly reads and writes model state while refresh preserves the volatile DRAM contents.
Science and assumptions
Parallel width creates aggregate bandwidth
Peak bandwidth equals active data-wire count times effective bit rate per wire divided by eight bits per byte.
Symbols and units
- B_peak
- Peak interface byte rate before losses
bytes/s
- W_data
- Number of active parallel data wires
wires
- r_pin
- Effective bit rate on each data wire
bits/(s·wire)
- 8
- Bits per byte
bits/byte
Wdata is the number of active data wires and rpin is the transfer rate in bits per second on each wire. The factor eight converts bits into bytes. Stacking more DRAM layers increases storage capacity, whereas interface width and signaling rate determine this bandwidth ceiling. These are separate design dimensions, explaining why a taller memory stack need not be a faster one.
Assumptions Ideal interface payload rate before command, refresh, bank-conflict, or scheduling losses. Count only data wires, and use a rate definition consistent with the signaling format. Capacity and bandwidth refer to a named stack configuration.
Engineering example
Adding a memory die can increase capacity but also raises alignment, thermal and test demands; interface width is a separate choice.
Distinctions and tradeoffs
Increasing the number of stacked DRAM layers primarily adds capacity. Bandwidth also depends on interface width and operating rate, so it does not automatically scale with height.
HBM offers exceptional bandwidth in a compact footprint but needs advanced stacking and packaging. Taller stacks increase integration difficulty and complicate heat removal.
Memory capacity determines which model states fit near the accelerator. Memory bandwidth determines how quickly those states can reach the arithmetic units.
Specifications and interfaces
- Micron HBM3E examples
- 24 GB 8-high / 36 GB 12-high
Vendor stack products, not a claim that both appear in GB200.
- Vendor stack bandwidth
- Greater than 1.2 TB/s
Micron HBM3E rated capability; GPU operating point can differ.
- Interface width
- 1,024 data bits
HBM3E architecture; excludes auxiliary and power connections.
Interfaces
- Read/write commands Input
- Requests through base interface
- Weight and state data Output
- Wide parallel transfers toward GPU
- Power/heat Input
- Electrical supply; heat leaves through package thermal paths
Check your understanding
What mainly distinguishes HBM’s approach from simply raising the speed of a narrow memory bus?
- HBM eliminates all DRAM refresh operations.
- It uses many parallel data connections close to the processor.
- It stores bits permanently without electrical power.
Answer and reasoning
It uses many parallel data connections close to the processor.
A very wide nearby interface provides high aggregate bandwidth. Stacking adds capacity in a compact footprint, while the memory remains volatile.
Historical context
2016 · Pascal P100 combines HBM2 packaging with NVLink
Announced on April 5, 2016, Tesla P100 brought HBM2 memory in a CoWoS package together with NVLink in NVIDIA’s Pascal accelerator platform. Faster local memory and GPU-to-GPU communication addressed data movement alongside arithmetic, anticipating the system bottlenecks of larger neural networks.
Evidence and scope
The eight-layer drawing is schematic and not a universal HBM generation.
Adjust stack layers, thickness, joint route, interface bandwidth and conditional yield; presets are synthetic, not supplier process windows.