The mechanism
The memory wall is a mismatch between useful computation and the ability to store or move its operands. A 2024 paper by Gholami and colleagues reports historical fitted trends for peak compute, DRAM bandwidth and single-GPU capacity. Its Figure 1 and Figure 2 use different samples and normalization, so their trends cannot be combined as though they measured the same population on the same basis. Such fits motivate investigating memory constraints but do not predict a particular future chip or prove that every AI kernel is memory-bound. A workload diagnosis instead measures operation count, transferred bytes, resident state and dependent-access latency at a clearly chosen boundary.
- Read each fitted metric with its source sample, time basis and normalization rather than merging several charts.
- Separate installed capacity from traffic rate and individual access latency in the workload under study.
- Measure operations and bytes crossing one chosen memory boundary to estimate arithmetic intensity.
- Compare that intensity with the machine’s compute-to-bandwidth balance under matching precision.
- Test the hypothesis with profiling, because launch overhead, dependencies and communication can sit below the simple bound.
Science and assumptions
Roofline bound
Attainable throughput cannot exceed the smaller of arithmetic peak and bandwidth times arithmetic intensity.
Symbols and units
- P
- Attainable throughput in the specified operation format
operations/s
- P_peak
- Compatible arithmetic ceiling
operations/s
- B
- Useful bandwidth at the selected memory boundary
bytes/s
- I
- Useful operations per byte crossing that boundary
operations/byte
P and Ppeak are operations/s, B is bytes/s and I is operations/byte at the same boundary. Precision and operation counting must match.
Assumptions A throughput ceiling; no guarantee of reaching it and no direct accounting for serial latency, launch overhead or cross-device synchronization.
Engineering example
Synthetic roofline: at 100 TOP/s and 1 TB/s, the ridge point is 100 operations/byte. A kernel at 10 operations/byte has a 10 TOP/s bandwidth ceiling even though the arithmetic units advertise ten times more.
Distinctions and tradeoffs
A historical fitted growth factor is not a physical law or a forecast, and a model fitting in memory can still be bandwidth-limited.
Adding arithmetic helps compute-limited work; increasing capacity can avoid offload without speeding each read; higher bandwidth helps only when enough independent transfers exist.
Training, prompt processing and low-batch autoregressive decode stress memory differently, so one historical chart cannot stand in for a phase-specific resource trace.
Specifications and interfaces
- Historical evidence
- Fitted ratios per two years
The ratio is dimensionless growth, not present-day throughput.
- Three constraints
- Capacity, bandwidth, latency
Bytes, bytes/s and seconds describe different limitations.
Interfaces
- Historical source sample Input
- Generation-specific observations and fitted rates
- Workload diagnosis Output
- Explicit capacity, traffic and latency hypotheses
Check your understanding
Why can a model fit in HBM yet run slowly?
- It may repeatedly move many bytes per useful operation
- Capacity guarantees peak arithmetic speed
- All memory accesses take zero time once allocated
Answer and reasoning
It may repeatedly move many bytes per useful operation
Fitting resolves a residency constraint; it does not guarantee sufficient traffic rate or low latency for the computation.
Evidence and scope
The three numerical historical fits appear only in separate claim records. Figure 2’s memory/6 plotting convention is not a universal training-state formula or an updated forecast.
Compare source-scoped normalized compute, capacity and bandwidth trends while retaining their different samples and denominators.