The mechanism
Memory alternatives solve different constraints. SRAM integrates near logic and supports fast local access, but has lower bit density than DRAM. HBM uses stacked DRAM and many short package connections. DDR commonly supplies expandable host memory; LPDDR emphasizes mobile-style power and signaling choices and also appears in some server and custom-accelerator designs. GDDR uses high-rate interfaces in discrete memory packages, providing another accelerator memory route. Capacity, sustained bandwidth, power, board area, package complexity and supply all matter. A valid comparison fixes a generation and configuration instead of treating one family as universally faster or cheaper.
- Determine which state must stay resident and which can be streamed without breaking the service target.
- Estimate reuse and concurrent accesses so the required bandwidth is tied to useful data movement.
- Evaluate local SRAM together with external DRAM rather than choosing one memory technology in isolation.
- Budget controller, PHY, package and board resources for the chosen width and signaling rate.
- Compare qualified configurations under the same workload while labeling price, supply and undocumented implementation details as unknown.
Science and assumptions
Raw interface rate
Raw interface bandwidth is active pin count times effective bit rate per pin divided by eight.
Symbols and units
- B_raw
- Raw directional interface byte rate
bytes/s
- w
- Active data-pin count
pins
- r
- Effective bit rate per active data pin
bits/(s·pin)
- 8
- Bits per byte
bits/byte
w is active data pins and r is effective bits/s per pin; Braw is bytes/s. Multiply by a measured efficiency to estimate usable bandwidth.
Assumptions Do not double an already effective DDR data rate; command overhead, refresh and bank conflicts reduce realized throughput.
Engineering example
Synthetic interfaces: 256 data pins at 16 Gbit/s and 1,024 pins at 4 Gbit/s both yield 512 GB/s raw. Their package requirements, power and latency can still be very different.
Distinctions and tradeoffs
HBM is not simply DDR stacked higher, and high pin rate alone does not determine aggregate usable bandwidth.
HBM’s wide nearby interface adds advanced package dependencies; discrete DRAM choices move more routing onto a board, while additional SRAM competes with compute area.
The right memory balance differs for a small reusable recommendation working set, a large model’s weights and rapidly growing request KV state.
Specifications and interfaces
- Interface bandwidth
- Data pins × bit rate / 8
Bytes/s before utilization losses; effective data rate already accounts for signaling transfers.
- Technology choice
- Memory device plus integration route
Controller/PHY, package, board and cooling must support the chosen interface.
Interfaces
- Workload memory demand Input
- Resident bytes, reuse, burst size and access concurrency
- Memory subsystem Output
- Delivered data under capacity, power and integration constraints
Check your understanding
Which change can raise aggregate bandwidth without raising per-pin rate?
- Increasing the number of simultaneously active data pins
- Increasing only stored capacity
- Renaming memory as cache
Answer and reasoning
Increasing the number of simultaneously active data pins
A wider interface transfers more bits in parallel, provided controllers, channels and traffic can use that width.
Evidence and scope
Pin counts and rates in the example are synthetic. No contemporary contract price, universal HBM advantage or interchangeable memory-controller compatibility is claimed.
Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.