{
  "schemaVersion": 1,
  "id": "atlas-lesson-memory-wall-trends",
  "title": "Historical memory-wall evidence",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Separates historical fitted compute, bandwidth and capacity trends from workload diagnosis.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/memory-wall-trends/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Custom silicon ecosystem",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The three numerical historical fits appear only in separate claim records. Figure 2’s memory/6 plotting convention is not a universal training-state formula or an updated forecast.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-wall-trends-109a75fba8791b9f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#memory-wall-trends",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "402b258f60e7e0eb29c326dc71926ca2d3d5a61dbc03899c813a6e2f34827225",
  "contentLength": 5443,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-wall-trends.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-wall-trends.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Historical memory-wall evidence",
      "start": 41,
      "end": 401
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 401,
      "end": 1669
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1669,
      "end": 1697
    },
    {
      "id": "section-4",
      "title": "Roofline bound",
      "start": 1697,
      "end": 2438
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2438,
      "end": 2671
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2671,
      "end": 3187
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3187,
      "end": 3444
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3444,
      "end": 3621
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3621,
      "end": 3650
    },
    {
      "id": "section-10",
      "title": "Why can a model fit in HBM yet run slowly?",
      "start": 3650,
      "end": 4060
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4060,
      "end": 4396
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4396,
      "end": 4860
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4860,
      "end": 4883
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4883,
      "end": 5098
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5098,
      "end": 5443
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#memory-wall-trends"
    },
    {
      "title": "AI and Memory Wall, version 1",
      "url": "https://arxiv.org/html/2403.14123v1"
    },
    {
      "title": "Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures",
      "url": "https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-wall-trends-109a75fba8791b9f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Full memory hierarchy and input path",
      "url": "https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/"
    },
    {
      "title": "Multiply–accumulate and fused arithmetic",
      "url": "https://semiaifoundry.com/learn/atlas/mac/"
    },
    {
      "title": "Training state and activation rematerialization",
      "url": "https://semiaifoundry.com/learn/atlas/training-memory-state/"
    },
    {
      "title": "Prefill and prompt processing",
      "url": "https://semiaifoundry.com/learn/atlas/prefill-resource-trace/"
    },
    {
      "title": "Autoregressive decode and weight traffic",
      "url": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/"
    }
  ],
  "recordHash": "318b96334bdbab6d345fffd29a843b24d60949419f038cb8d3958ebcc8063bf0",
  "contentFormat": "text/markdown",
  "content": "Custom silicon ecosystem · Atlas v1.3.0\n\n# Historical memory-wall evidence\n\nSeparates historical fitted compute, bandwidth and capacity trends from workload diagnosis.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-wall-trends)Register to storage hierarchy and product lifecycle\n\nLearning objectiveUse historical trends as scoped evidence and diagnose a concrete workload separately.\n\n## The mechanism\n\nThe memory wall is a mismatch between useful computation and the ability to store or move its operands. A 2024 paper by Gholami and colleagues reports historical fitted trends for peak compute, DRAM bandwidth and single-GPU capacity. Its Figure 1 and Figure 2 use different samples and normalization, so their trends cannot be combined as though they measured the same population on the same basis. Such fits motivate investigating memory constraints but do not predict a particular future chip or prove that every AI kernel is memory-bound. A workload diagnosis instead measures operation count, transferred bytes, resident state and dependent-access latency at a clearly chosen boundary.\n\n1.  Read each fitted metric with its source sample, time basis and normalization rather than merging several charts.\n2.  Separate installed capacity from traffic rate and individual access latency in the workload under study.\n3.  Measure operations and bytes crossing one chosen memory boundary to estimate arithmetic intensity.\n4.  Compare that intensity with the machine’s compute-to-bandwidth balance under matching precision.\n5.  Test the hypothesis with profiling, because launch overhead, dependencies and communication can sit below the simple bound.\n\n## Science and assumptions\n\n### Roofline bound\n\n$$\nP\\le\\min(P_{\\mathrm{peak}},BI)\n$$\n\nAttainable throughput cannot exceed the smaller of arithmetic peak and bandwidth times arithmetic intensity.\n\nSymbols and units\n\nP\n\nAttainable throughput in the specified operation format\n\noperations/s\n\nP\\_peak\n\nCompatible arithmetic ceiling\n\noperations/s\n\nB\n\nUseful bandwidth at the selected memory boundary\n\nbytes/s\n\nI\n\nUseful operations per byte crossing that boundary\n\noperations/byte\n\nP and Ppeak are operations/s, B is bytes/s and I is operations/byte at the same boundary. Precision and operation counting must match.\n\nAssumptions A throughput ceiling; no guarantee of reaching it and no direct accounting for serial latency, launch overhead or cross-device synchronization.\n\n## Engineering example\n\nSynthetic roofline: at 100 TOP/s and 1 TB/s, the ridge point is 100 operations/byte. A kernel at 10 operations/byte has a 10 TOP/s bandwidth ceiling even though the arithmetic units advertise ten times more.\n\n## Distinctions and tradeoffs\n\nA historical fitted growth factor is not a physical law or a forecast, and a model fitting in memory can still be bandwidth-limited.\n\nAdding arithmetic helps compute-limited work; increasing capacity can avoid offload without speeding each read; higher bandwidth helps only when enough independent transfers exist.\n\nTraining, prompt processing and low-batch autoregressive decode stress memory differently, so one historical chart cannot stand in for a phase-specific resource trace.\n\n## Specifications and interfaces\n\nHistorical evidence\n\nFitted ratios per two years\n\nThe ratio is dimensionless growth, not present-day throughput.\n\nThree constraints\n\nCapacity, bandwidth, latency\n\nBytes, bytes/s and seconds describe different limitations.\n\n### Interfaces\n\nHistorical source sample Input\n\nGeneration-specific observations and fitted rates\n\nWorkload diagnosis Output\n\nExplicit capacity, traffic and latency hypotheses\n\n## Check your understanding\n\n### Why can a model fit in HBM yet run slowly?\n\n1.  It may repeatedly move many bytes per useful operation\n2.  Capacity guarantees peak arithmetic speed\n3.  All memory accesses take zero time once allocated\n\nAnswer and reasoning\n\nIt may repeatedly move many bytes per useful operation\n\nFitting resolves a residency constraint; it does not guarantee sufficient traffic rate or low latency for the computation.\n\n## Evidence and scope\n\nThe three numerical historical fits appear only in separate claim records. Figure 2’s memory/6 plotting convention is not a universal training-state formula or an updated forecast.\n\nCompare source-scoped normalized compute, capacity and bandwidth trends while retaining their different samples and denominators.\n\n## Sources\n\n-   [AI and Memory Wall, version 1](https://arxiv.org/html/2403.14123v1)\n-   [Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-134.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-wall-trends-109a75fba8791b9f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Full memory hierarchy and input path](https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/)\n-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n\n### Related lessons\n\n-   [Training state and activation rematerialization](https://semiaifoundry.com/learn/atlas/training-memory-state/)\n-   [Prefill and prompt processing](https://semiaifoundry.com/learn/atlas/prefill-resource-trace/)\n-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)"
}