{"id":"hbm-controller","level":"die","name":"HBM controller and PHY","shortName":"HBM interface","role":"Schedules DRAM commands and translates on-chip requests into the electrical signals of the memory interface.","description":"A memory controller turns requests for addresses into commands that the DRAM organization can execute. It coordinates reads, writes, timing constraints, and maintenance such as refresh. The PHY is the physical interface circuitry that drives and receives signals across package wires. These are distinct jobs: the controller manages the protocol and schedule, while the PHY makes the bits electrically reliable. Many parallel wires provide aggregate HBM bandwidth without demanding that each wire run at the highest possible rate. A request passes through controller queues and bank selection before its data crosses the memory interface and package wiring. Read responses return through that path to the requesting work. Published stack bandwidth is an upper-bound interface capability, not the sustained bandwidth of every access pattern.","specs":[{"label":"HBM3E interface width","value":"1,024 data bits per stack","context":"HBM3E wide-interface organization; excludes command, clock, power, and ground connections."},{"label":"Reference GPU memory bandwidth","value":"8 TB/s per GPU","context":"Derived from the GB200 product table’s 16 TB/s per two-GPU Superchip."},{"label":"Memory access","value":"Read/write commands plus refresh","context":"Protocol scheduling and physical signaling are separate functions."}],"interfaces":[{"name":"Memory requests","direction":"in","signal":"Addresses and data from L2/fabric"},{"name":"DRAM commands","direction":"out","signal":"Bank/row/column operations and writes"},{"name":"Read data","direction":"in","signal":"Parallel data from HBM PHY"}],"tradeoff":"Wider interfaces increase bandwidth but require dense package routing and more connections. Poorly localized or conflicting accesses can achieve much less than the aggregate peak.","whyAI":"Decoding a token often streams many model weights, so memory delivery can dominate latency. HBM controllers connect large model state to the much smaller arithmetic neighborhoods.","related":["hbm-stack","interposer","l2-cache","on-chip-network"],"sources":[{"title":"Micron HBM3E product brief","url":"https://assets.micron.com/adobe/assets/urn%3Aaaid%3Aaem%3Ab710d8f2-7f66-44c1-a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf"},{"title":"NVIDIA GB200 NVL72 specifications","url":"https://www.nvidia.com/en-us/data-center/gb200-nvl72/"},{"title":"Intel: Memory performance, latency, and bandwidth","url":"https://www.intel.com/content/www/us/en/developer/articles/technical/memory-performance-in-a-nutshell.html"},{"title":"John D. C. Little (1961): A Proof for the Queuing Formula L = λW","url":"https://pubsonline.informs.org/doi/abs/10.1287/opre.9.3.383"}],"scale":"Die-edge interface blocks and millimeter package links; schematic layout","kind":"controller-queue-and-bus","mechanism":["The controller accepts memory requests, maps addresses to memory resources, and schedules legal operations around timing constraints.","The PHY drives or receives many electrical signals across the package interface to the selected HBM stack.","Read data return to the cache and fabric, while refresh and other maintenance continue alongside application traffic."],"misconception":"Rated aggregate HBM bandwidth is not the speed of every individual access. Bank conflicts, transfer patterns, and protocol timing affect sustained throughput.","check":{"question":"What is the difference between the controller and the PHY?","options":["The controller schedules memory operations; the PHY implements electrical signaling.","The controller stores all weights; the PHY performs attention.","They are two names for an NVMe drive."],"answer":0,"explanation":"Protocol scheduling and physical bit transmission are distinct functions, even when integrated into one memory-interface subsystem."},"branch":"Memory hierarchy","subbranch":"DRAM interfaces","science":[{"title":"Concurrent requests cover memory latency","equation":"Nneeded ≈ Btarget × L / s","explanation":"Btarget is the desired useful bandwidth in bytes per second, L is average request latency in seconds, and s is useful bytes per completed request. The result estimates how many independent requests must be outstanding. Sequentially dependent loads may fail to expose that concurrency even when the memory stack advertises ample bandwidth; controller queues cannot create independence absent from the program.","assumptions":"Steady-state Little’s-law estimate with a consistent request size and measured latency. Bank timing, refresh, arbitration, and protocol overhead determine the achievable B and L; no vendor queue capacity is assumed."}],"history":[],"learningObjective":"Translate a bandwidth target into a required interface count.","engineeringExample":"For a target byte rate B and per-link latency L, N_needed≈BL/s is a sizing relation; channel scheduling and PHY overhead remain.","evidenceNotes":"It is a dimensional estimate, not a product channel count.","visualFamily":"memory-route","prerequisites":["dram-die","interconnect"],"labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}