# Historical memory-wall evidence

Canonical: https://semiaifoundry.com/learn/atlas/memory-wall-trends/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Custom silicon ecosystem · Atlas v1.3.0

# Historical memory-wall evidence

Separates historical fitted compute, bandwidth and capacity trends from workload diagnosis.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-wall-trends)Register to storage hierarchy and product lifecycle

Learning objectiveUse historical trends as scoped evidence and diagnose a concrete workload separately.

## The mechanism

The memory wall is a mismatch between useful computation and the ability to store or move its operands. A 2024 paper by Gholami and colleagues reports historical fitted trends for peak compute, DRAM bandwidth and single-GPU capacity. Its Figure 1 and Figure 2 use different samples and normalization, so their trends cannot be combined as though they measured the same population on the same basis. Such fits motivate investigating memory constraints but do not predict a particular future chip or prove that every AI kernel is memory-bound. A workload diagnosis instead measures operation count, transferred bytes, resident state and dependent-access latency at a clearly chosen boundary.

1.  Read each fitted metric with its source sample, time basis and normalization rather than merging several charts.
2.  Separate installed capacity from traffic rate and individual access latency in the workload under study.
3.  Measure operations and bytes crossing one chosen memory boundary to estimate arithmetic intensity.
4.  Compare that intensity with the machine’s compute-to-bandwidth balance under matching precision.
5.  Test the hypothesis with profiling, because launch overhead, dependencies and communication can sit below the simple bound.

## Science and assumptions

### Roofline bound

$$
P\le\min(P_{\mathrm{peak}},BI)
$$

Attainable throughput cannot exceed the smaller of arithmetic peak and bandwidth times arithmetic intensity.

Symbols and units

P

Attainable throughput in the specified operation format

operations/s

P\_peak

Compatible arithmetic ceiling

operations/s

B

Useful bandwidth at the selected memory boundary

bytes/s

I

Useful operations per byte crossing that boundary

operations/byte

P and Ppeak are operations/s, B is bytes/s and I is operations/byte at the same boundary. Precision and operation counting must match.

Assumptions A throughput ceiling; no guarantee of reaching it and no direct accounting for serial latency, launch overhead or cross-device synchronization.

## Engineering example

Synthetic roofline: at 100 TOP/s and 1 TB/s, the ridge point is 100 operations/byte. A kernel at 10 operations/byte has a 10 TOP/s bandwidth ceiling even though the arithmetic units advertise ten times more.

## Distinctions and tradeoffs

A historical fitted growth factor is not a physical law or a forecast, and a model fitting in memory can still be bandwidth-limited.

Adding arithmetic helps compute-limited work; increasing capacity can avoid offload without speeding each read; higher bandwidth helps only when enough independent transfers exist.

Training, prompt processing and low-batch autoregressive decode stress memory differently, so one historical chart cannot stand in for a phase-specific resource trace.

## Specifications and interfaces

Historical evidence

Fitted ratios per two years

The ratio is dimensionless growth, not present-day throughput.

Three constraints

Capacity, bandwidth, latency

Bytes, bytes/s and seconds describe different limitations.

### Interfaces

Historical source sample Input

Generation-specific observations and fitted rates

Workload diagnosis Output

Explicit capacity, traffic and latency hypotheses

## Check your understanding

### Why can a model fit in HBM yet run slowly?

1.  It may repeatedly move many bytes per useful operation
2.  Capacity guarantees peak arithmetic speed
3.  All memory accesses take zero time once allocated

Answer and reasoning

It may repeatedly move many bytes per useful operation

Fitting resolves a residency constraint; it does not guarantee sufficient traffic rate or low latency for the computation.

## Evidence and scope

The three numerical historical fits appear only in separate claim records. Figure 2’s memory/6 plotting convention is not a universal training-state formula or an updated forecast.

Compare source-scoped normalized compute, capacity and bandwidth trends while retaining their different samples and denominators.

## Sources

-   [AI and Memory Wall, version 1](https://arxiv.org/html/2403.14123v1)
-   [Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-wall-trends-109a75fba8791b9f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Full memory hierarchy and input path](https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/)
-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)

### Related lessons

-   [Training state and activation rematerialization](https://semiaifoundry.com/learn/atlas/training-memory-state/)
-   [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)
-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)