{
  "schemaVersion": 1,
  "id": "atlas-lesson-l2-cache",
  "title": "Shared L2 cache",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Provides a larger on-chip reuse layer between SM traffic and HBM.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/l2-cache/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Memory hierarchy",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "AMAT is an average model and does not capture overlap or queue contention.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/l2-cache-26b3463fc4abe144.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#l2-cache",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "9c3917d4a3b1c2be736adae2304f9b7941234ab701837462ae9a71bacdd8de5a",
  "contentLength": 5366,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-l2-cache.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-l2-cache.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Shared L2 cache",
      "start": 33,
      "end": 297
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 297,
      "end": 1362
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1362,
      "end": 1390
    },
    {
      "id": "section-4",
      "title": "Average access cost across two cache levels",
      "start": 1390,
      "end": 2508
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2508,
      "end": 2649
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2649,
      "end": 3190
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3190,
      "end": 3514
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3514,
      "end": 3711
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3711,
      "end": 3740
    },
    {
      "id": "section-10",
      "title": "When does a larger L2 cache most directly help?",
      "start": 3740,
      "end": 4165
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4165,
      "end": 4417
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4417,
      "end": 5055
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5055,
      "end": 5078
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5078,
      "end": 5172
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5172,
      "end": 5366
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#l2-cache"
    },
    {
      "title": "NVIDIA CUDA Blackwell Tuning Guide",
      "url": "https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html"
    },
    {
      "title": "NVIDIA: GPU locality domains and L2 fabric",
      "url": "https://developer.nvidia.com/blog/accelerating-data-processing-with-nvidia-multi-instance-gpu-and-numa-node-localization/"
    },
    {
      "title": "MIT Computation Structures: cache hierarchy and average access time",
      "url": "https://ocw.mit.edu/courses/6-004-computation-structures-spring-2017/pages/c14/c14s1/"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/l2-cache-26b3463fc4abe144.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "L1 and texture cache",
      "url": "https://semiaifoundry.com/learn/atlas/l1-cache/"
    },
    {
      "title": "On-chip interconnect fabric",
      "url": "https://semiaifoundry.com/learn/atlas/on-chip-network/"
    },
    {
      "title": "HBM controller and PHY",
      "url": "https://semiaifoundry.com/learn/atlas/hbm-controller/"
    }
  ],
  "recordHash": "41c658a10bf525907f25c5293861dca853584fc514e9b62f055e78af4ae16573",
  "contentFormat": "text/markdown",
  "content": "Memory hierarchy · Atlas v1.3.0\n\n# Shared L2 cache\n\nProvides a larger on-chip reuse layer between SM traffic and HBM.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#l2-cache)Distributed banks across a centimeter-scale GPU die\n\nLearning objectiveUse AMAT to expose nested miss penalties.\n\n## The mechanism\n\nL2 sits farther from individual execution units and serves requests across the GPU. A tile loaded by one part of the chip can remain available for later work, reducing traffic to external HBM. The cache is physically distributed even when software sees one device address space. Address placement and work placement can therefore influence the distance data travels. Persistence controls can help protect selected reuse, but they do not make the entire model fit into cache. A useful view shows multiple SMs converging on cache slices, with a counter for bytes served locally versus bytes fetched from HBM. Cache capacity should be compared with a working set, not confused with total model storage.\n\n1.  Requests that cannot be satisfied close to an SM reach shared L2 cache slices through the internal fabric.\n2.  An L2 hit returns retained data; a miss asks the memory controllers to fetch data from HBM.\n3.  Reuse by later work can avoid additional HBM trips, while replacement policies and working-set size determine what remains available.\n\n## Science and assumptions\n\n### Average access cost across two cache levels\n\n$$\n\\mathrm{AMAT}=t_{L1}+m_{L1}(t_{L2}+m_{L2}t_{\\mathrm{HBM}})\n$$\n\nAverage access time equals L1 lookup time plus the L1 miss fraction times L2 lookup time and its conditional HBM miss penalty.\n\nSymbols and units\n\nAMAT\n\nAverage memory access time in the sequential model\n\ns\n\nt\\_L1\n\nL1 lookup time\n\ns\n\nm\\_L1\n\nFraction of accesses missing L1\n\nfraction\n\nt\\_L2\n\nAdditional L2 lookup time after an L1 miss\n\ns\n\nm\\_L2\n\nL2 miss fraction conditional on an L1 miss\n\nfraction\n\nt\\_HBM\n\nAdditional service time after an L2 miss\n\ns\n\ntL1 and tL2 are lookup times; mL1 is the fraction missing in L1 and mL2 is the fraction of those requests that also miss in L2. tHBM is the additional service time after an L2 miss. Lowering mL2 prevents some expensive trips to HBM, making a cache useful even when it stores only a small portion of total device memory.\n\nAssumptions A sequential lookup model with conditional miss rates and incremental latency terms. Real GPUs overlap requests, queue transactions, and hide latency with other work, so AMAT is not the same as kernel execution time.\n\n## Engineering example\n\nA 5% L1 miss rate makes the L2 path visible; a 20% L2 miss rate then exposes the HBM term in the nested expression.\n\n## Distinctions and tradeoffs\n\nA unified device address space does not mean every physical memory route has identical cost. Distributed cache and internal locality can still matter.\n\nA larger cache can save external bandwidth but consumes expensive logic-die area. Locality and scheduling can matter as much as nominal capacity when requests cross internal partitions.\n\nReused weights and intermediate tensors can be served without another HBM access. This helps only when the active working set and access pattern fit the cache hierarchy.\n\n## Specifications and interfaces\n\nPublished capacity\n\n126 MB\n\nNVIDIA Blackwell Tuning Guide labels this as GB200 GPU L2 capacity.\n\nScope\n\nShared device cache\n\nPhysically distributed; not private storage for each SM.\n\nReuse control\n\nL2 persistence hints\n\nCUDA policies influence retention; they are not capacity guarantees.\n\n### Interfaces\n\nSM requests Input\n\nLoads, stores, and atomic operations\n\nCache response Output\n\nReused data to requesting compute units\n\nMiss/refill Output\n\nTransactions to HBM memory controllers\n\n## Check your understanding\n\n### When does a larger L2 cache most directly help?\n\n1.  Whenever a model has more vocabulary entries, regardless of access.\n2.  Only when the network cable is unplugged.\n3.  When useful data are reused before they are evicted.\n\nAnswer and reasoning\n\nWhen useful data are reused before they are evicted.\n\nA cache reduces external traffic when it retains data that will be needed again. Capacity alone does not create reuse.\n\n## Evidence and scope\n\nAMAT is an average model and does not capture overlap or queue contention.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [NVIDIA CUDA Blackwell Tuning Guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html)\n-   [NVIDIA: GPU locality domains and L2 fabric](https://developer.nvidia.com/blog/accelerating-data-processing-with-nvidia-multi-instance-gpu-and-numa-node-localization/)\n-   [MIT Computation Structures: cache hierarchy and average access time](https://ocw.mit.edu/courses/6-004-computation-structures-spring-2017/pages/c14/c14s1/)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/l2-cache-26b3463fc4abe144.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [L1 and texture cache](https://semiaifoundry.com/learn/atlas/l1-cache/)\n\n### Related lessons\n\n-   [On-chip interconnect fabric](https://semiaifoundry.com/learn/atlas/on-chip-network/)\n-   [HBM controller and PHY](https://semiaifoundry.com/learn/atlas/hbm-controller/)"
}