HBM controller and PHY

The mechanism

A memory controller turns requests for addresses into commands that the DRAM organization can execute. It coordinates reads, writes, timing constraints, and maintenance such as refresh. The PHY is the physical interface circuitry that drives and receives signals across package wires. These are distinct jobs: the controller manages the protocol and schedule, while the PHY makes the bits electrically reliable. Many parallel wires provide aggregate HBM bandwidth without demanding that each wire run at the highest possible rate. A request passes through controller queues and bank selection before its data crosses the memory interface and package wiring. Read responses return through that path to the requesting work. Published stack bandwidth is an upper-bound interface capability, not the sustained bandwidth of every access pattern.

The controller accepts memory requests, maps addresses to memory resources, and schedules legal operations around timing constraints.
The PHY drives or receives many electrical signals across the package interface to the selected HBM stack.
Read data return to the cache and fabric, while refresh and other maintenance continue alongside application traffic.

Science and assumptions

Concurrent requests cover memory latency

N_{\mathrm{needed}}\approx\frac{B_{\mathrm{target}}L}{s}

Required request concurrency is approximately target bandwidth times latency divided by useful bytes per request.

Symbols and units

N_needed: Average independent requests needed in flight
requests
B_target: Target useful byte rate
bytes/s
L: Mean request latency
s
s: Useful bytes per completed request
bytes/request

Btarget is the desired useful bandwidth in bytes per second, L is average request latency in seconds, and s is useful bytes per completed request. The result estimates how many independent requests must be outstanding. Sequentially dependent loads may fail to expose that concurrency even when the memory stack advertises ample bandwidth; controller queues cannot create independence absent from the program.

Assumptions Steady-state Little’s-law estimate with a consistent request size and measured latency. Bank timing, refresh, arbitration, and protocol overhead determine the achievable B and L; no vendor queue capacity is assumed.

Engineering example

For a target byte rate B and per-link latency L, $N_{\mathrm{needed}}≈BL/s$ is a sizing relation; channel scheduling and PHY overhead remain.

Distinctions and tradeoffs

Rated aggregate HBM bandwidth is not the speed of every individual access. Bank conflicts, transfer patterns, and protocol timing affect sustained throughput.

Wider interfaces increase bandwidth but require dense package routing and more connections. Poorly localized or conflicting accesses can achieve much less than the aggregate peak.

Decoding a token often streams many model weights, so memory delivery can dominate latency. HBM controllers connect large model state to the much smaller arithmetic neighborhoods.

Specifications and interfaces

HBM3E interface width: 1,024 data bits per stack
HBM3E wide-interface organization; excludes command, clock, power, and ground connections.
Reference GPU memory bandwidth: 8 TB/s per GPU
Derived from the GB200 product table’s 16 TB/s per two-GPU Superchip.
Memory access: Read/write commands plus refresh
Protocol scheduling and physical signaling are separate functions.

Interfaces

Memory requests Input: Addresses and data from L2/fabric
DRAM commands Output: Bank/row/column operations and writes
Read data Input: Parallel data from HBM PHY

Check your understanding

What is the difference between the controller and the PHY?

The controller schedules memory operations; the PHY implements electrical signaling.
The controller stores all weights; the PHY performs attention.
They are two names for an NVMe drive.

Answer and reasoning

The controller schedules memory operations; the PHY implements electrical signaling.

Protocol scheduling and physical bit transmission are distinct functions, even when integrated into one memory-interface subsystem.

Evidence and scope

It is a dimensional estimate, not a product channel count.

Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.