{"id":"memory-wall-trends","level":"compute","name":"Historical memory-wall evidence","shortName":"Historical memory-wall evidence","role":"Separates historical fitted compute, bandwidth and capacity trends from workload diagnosis.","description":"The memory wall is a mismatch between useful computation and the ability to store or move its operands. A 2024 paper by Gholami and colleagues reports historical fitted trends for peak compute, DRAM bandwidth and single-GPU capacity. Its Figure 1 and Figure 2 use different samples and normalization, so their trends cannot be combined as though they measured the same population on the same basis. Such fits motivate investigating memory constraints but do not predict a particular future chip or prove that every AI kernel is memory-bound. A workload diagnosis instead measures operation count, transferred bytes, resident state and dependent-access latency at a clearly chosen boundary.","specs":[{"label":"Historical evidence","value":"Fitted ratios per two years","context":"The ratio is dimensionless growth, not present-day throughput."},{"label":"Three constraints","value":"Capacity, bandwidth, latency","context":"Bytes, bytes/s and seconds describe different limitations."}],"interfaces":[{"name":"Historical source sample","direction":"in","signal":"Generation-specific observations and fitted rates"},{"name":"Workload diagnosis","direction":"out","signal":"Explicit capacity, traffic and latency hypotheses"}],"tradeoff":"Adding arithmetic helps compute-limited work; increasing capacity can avoid offload without speeding each read; higher bandwidth helps only when enough independent transfers exist.","whyAI":"Training, prompt processing and low-batch autoregressive decode stress memory differently, so one historical chart cannot stand in for a phase-specific resource trace.","related":["training-memory-state","prefill-resource-trace","decode-bandwidth-trace"],"sources":[{"title":"AI and Memory Wall, version 1","url":"https://arxiv.org/html/2403.14123v1"},{"title":"Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures","url":"https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html"}],"scale":"Register to storage hierarchy and product lifecycle","kind":"architecture","branch":"Custom silicon ecosystem","subbranch":"Memory choices and silicon economics","mechanism":["Read each fitted metric with its source sample, time basis and normalization rather than merging several charts.","Separate installed capacity from traffic rate and individual access latency in the workload under study.","Measure operations and bytes crossing one chosen memory boundary to estimate arithmetic intensity.","Compare that intensity with the machine’s compute-to-bandwidth balance under matching precision.","Test the hypothesis with profiling, because launch overhead, dependencies and communication can sit below the simple bound."],"misconception":"A historical fitted growth factor is not a physical law or a forecast, and a model fitting in memory can still be bandwidth-limited.","check":{"question":"Why can a model fit in HBM yet run slowly?","options":["It may repeatedly move many bytes per useful operation","Capacity guarantees peak arithmetic speed","All memory accesses take zero time once allocated"],"answer":0,"explanation":"Fitting resolves a residency constraint; it does not guarantee sufficient traffic rate or low latency for the computation."},"science":[{"title":"Roofline bound","equation":"P ≤ min(Ppeak, B × I)","explanation":"P and Ppeak are operations/s, B is bytes/s and I is operations/byte at the same boundary. Precision and operation counting must match.","assumptions":"A throughput ceiling; no guarantee of reaching it and no direct accounting for serial latency, launch overhead or cross-device synchronization."}],"prerequisites":["memory-hierarchy-route","mac"],"learningObjective":"Use historical trends as scoped evidence and diagnose a concrete workload separately.","workstream":"W11","visualFamily":"memory-route","engineeringExample":"Synthetic roofline: at 100 TOP/s and 1 TB/s, the ridge point is 100 operations/byte. A kernel at 10 operations/byte has a 10 TOP/s bandwidth ceiling even though the arithmetic units advertise ten times more.","evidenceNotes":"The three numerical historical fits appear only in separate claim records. Figure 2’s memory/6 plotting convention is not a universal training-state formula or an updated forecast.","labId":"engineering","scope":"Compare source-scoped normalized compute, capacity and bandwidth trends while retaining their different samples and denominators.","deepLabId":"memory-growth","investigationId":"execution","labScope":"Compare source-scoped normalized compute, capacity and bandwidth trends while retaining their different samples and denominators."}