# HBM controller and PHY Canonical: https://semiaifoundry.com/learn/atlas/hbm-controller/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Memory hierarchy · Atlas v1.3.0 # HBM controller and PHY Schedules DRAM commands and translates on-chip requests into the electrical signals of the memory interface. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#hbm-controller)Die-edge interface blocks and millimeter package links; schematic layout Learning objectiveTranslate a bandwidth target into a required interface count. ## The mechanism A memory controller turns requests for addresses into commands that the DRAM organization can execute. It coordinates reads, writes, timing constraints, and maintenance such as refresh. The PHY is the physical interface circuitry that drives and receives signals across package wires. These are distinct jobs: the controller manages the protocol and schedule, while the PHY makes the bits electrically reliable. Many parallel wires provide aggregate HBM bandwidth without demanding that each wire run at the highest possible rate. A request passes through controller queues and bank selection before its data crosses the memory interface and package wiring. Read responses return through that path to the requesting work. Published stack bandwidth is an upper-bound interface capability, not the sustained bandwidth of every access pattern. 1. The controller accepts memory requests, maps addresses to memory resources, and schedules legal operations around timing constraints. 2. The PHY drives or receives many electrical signals across the package interface to the selected HBM stack. 3. Read data return to the cache and fabric, while refresh and other maintenance continue alongside application traffic. ## Science and assumptions ### Concurrent requests cover memory latency $$ N_{\mathrm{needed}}\approx\frac{B_{\mathrm{target}}L}{s} $$ Required request concurrency is approximately target bandwidth times latency divided by useful bytes per request. Symbols and units N\_needed Average independent requests needed in flight requests B\_target Target useful byte rate bytes/s L Mean request latency s s Useful bytes per completed request bytes/request Btarget is the desired useful bandwidth in bytes per second, L is average request latency in seconds, and s is useful bytes per completed request. The result estimates how many independent requests must be outstanding. Sequentially dependent loads may fail to expose that concurrency even when the memory stack advertises ample bandwidth; controller queues cannot create independence absent from the program. Assumptions Steady-state Little’s-law estimate with a consistent request size and measured latency. Bank timing, refresh, arbitration, and protocol overhead determine the achievable B and L; no vendor queue capacity is assumed. ## Engineering example For a target byte rate B and per-link latency L, $N_{\mathrm{needed}}≈BL/s$ is a sizing relation; channel scheduling and PHY overhead remain. ## Distinctions and tradeoffs Rated aggregate HBM bandwidth is not the speed of every individual access. Bank conflicts, transfer patterns, and protocol timing affect sustained throughput. Wider interfaces increase bandwidth but require dense package routing and more connections. Poorly localized or conflicting accesses can achieve much less than the aggregate peak. Decoding a token often streams many model weights, so memory delivery can dominate latency. HBM controllers connect large model state to the much smaller arithmetic neighborhoods. ## Specifications and interfaces HBM3E interface width 1,024 data bits per stack HBM3E wide-interface organization; excludes command, clock, power, and ground connections. Reference GPU memory bandwidth 8 TB/s per GPU Derived from the GB200 product table’s 16 TB/s per two-GPU Superchip. Memory access Read/write commands plus refresh Protocol scheduling and physical signaling are separate functions. ### Interfaces Memory requests Input Addresses and data from L2/fabric DRAM commands Output Bank/row/column operations and writes Read data Input Parallel data from HBM PHY ## Check your understanding ### What is the difference between the controller and the PHY? 1. The controller schedules memory operations; the PHY implements electrical signaling. 2. The controller stores all weights; the PHY performs attention. 3. They are two names for an NVMe drive. Answer and reasoning The controller schedules memory operations; the PHY implements electrical signaling. Protocol scheduling and physical bit transmission are distinct functions, even when integrated into one memory-interface subsystem. ## Evidence and scope It is a dimensional estimate, not a product channel count. Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism. ## Sources - [Micron HBM3E product brief](https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3Ab710d8f2-7f66-44c1-a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf) - [NVIDIA GB200 NVL72 specifications](https://www.nvidia.com/en-us/data-center/gb200-nvl72/) - [Intel: Memory performance, latency, and bandwidth](https://www.intel.com/content/www/us/en/developer/articles/technical/memory-performance-in-a-nutshell.html) - [John D. C. Little (1961): A Proof for the Queuing Formula L = λW](https://pubsonline.informs.org/doi/abs/10.1287/opre.9.3.383) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/hbm-controller-bf096ea73b5c5207.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [DRAM die and memory banks](https://semiaifoundry.com/learn/atlas/dram-die/) - [On-chip metal and via hierarchy](https://semiaifoundry.com/learn/atlas/interconnect/) ### Related lessons - [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/) - [Interposer and redistribution layers](https://semiaifoundry.com/learn/atlas/interposer/) - [Shared L2 cache](https://semiaifoundry.com/learn/atlas/l2-cache/) - [On-chip interconnect fabric](https://semiaifoundry.com/learn/atlas/on-chip-network/)