# Shared L2 cache Canonical: https://semiaifoundry.com/learn/atlas/l2-cache/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Memory hierarchy · Atlas v1.3.0 # Shared L2 cache Provides a larger on-chip reuse layer between SM traffic and HBM. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#l2-cache)Distributed banks across a centimeter-scale GPU die Learning objectiveUse AMAT to expose nested miss penalties. ## The mechanism L2 sits farther from individual execution units and serves requests across the GPU. A tile loaded by one part of the chip can remain available for later work, reducing traffic to external HBM. The cache is physically distributed even when software sees one device address space. Address placement and work placement can therefore influence the distance data travels. Persistence controls can help protect selected reuse, but they do not make the entire model fit into cache. A useful view shows multiple SMs converging on cache slices, with a counter for bytes served locally versus bytes fetched from HBM. Cache capacity should be compared with a working set, not confused with total model storage. 1. Requests that cannot be satisfied close to an SM reach shared L2 cache slices through the internal fabric. 2. An L2 hit returns retained data; a miss asks the memory controllers to fetch data from HBM. 3. Reuse by later work can avoid additional HBM trips, while replacement policies and working-set size determine what remains available. ## Science and assumptions ### Average access cost across two cache levels $$ \mathrm{AMAT}=t_{L1}+m_{L1}(t_{L2}+m_{L2}t_{\mathrm{HBM}}) $$ Average access time equals L1 lookup time plus the L1 miss fraction times L2 lookup time and its conditional HBM miss penalty. Symbols and units AMAT Average memory access time in the sequential model s t\_L1 L1 lookup time s m\_L1 Fraction of accesses missing L1 fraction t\_L2 Additional L2 lookup time after an L1 miss s m\_L2 L2 miss fraction conditional on an L1 miss fraction t\_HBM Additional service time after an L2 miss s tL1 and tL2 are lookup times; mL1 is the fraction missing in L1 and mL2 is the fraction of those requests that also miss in L2. tHBM is the additional service time after an L2 miss. Lowering mL2 prevents some expensive trips to HBM, making a cache useful even when it stores only a small portion of total device memory. Assumptions A sequential lookup model with conditional miss rates and incremental latency terms. Real GPUs overlap requests, queue transactions, and hide latency with other work, so AMAT is not the same as kernel execution time. ## Engineering example A 5% L1 miss rate makes the L2 path visible; a 20% L2 miss rate then exposes the HBM term in the nested expression. ## Distinctions and tradeoffs A unified device address space does not mean every physical memory route has identical cost. Distributed cache and internal locality can still matter. A larger cache can save external bandwidth but consumes expensive logic-die area. Locality and scheduling can matter as much as nominal capacity when requests cross internal partitions. Reused weights and intermediate tensors can be served without another HBM access. This helps only when the active working set and access pattern fit the cache hierarchy. ## Specifications and interfaces Published capacity 126 MB NVIDIA Blackwell Tuning Guide labels this as GB200 GPU L2 capacity. Scope Shared device cache Physically distributed; not private storage for each SM. Reuse control L2 persistence hints CUDA policies influence retention; they are not capacity guarantees. ### Interfaces SM requests Input Loads, stores, and atomic operations Cache response Output Reused data to requesting compute units Miss/refill Output Transactions to HBM memory controllers ## Check your understanding ### When does a larger L2 cache most directly help? 1. Whenever a model has more vocabulary entries, regardless of access. 2. Only when the network cable is unplugged. 3. When useful data are reused before they are evicted. Answer and reasoning When useful data are reused before they are evicted. A cache reduces external traffic when it retains data that will be needed again. Capacity alone does not create reuse. ## Evidence and scope AMAT is an average model and does not capture overlap or queue contention. Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism. ## Sources - [NVIDIA CUDA Blackwell Tuning Guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html) - [NVIDIA: GPU locality domains and L2 fabric](https://developer.nvidia.com/blog/accelerating-data-processing-with-nvidia-multi-instance-gpu-and-numa-node-localization/) - [MIT Computation Structures: cache hierarchy and average access time](https://ocw.mit.edu/courses/6-004-computation-structures-spring-2017/pages/c14/c14s1/) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/l2-cache-26b3463fc4abe144.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [L1 and texture cache](https://semiaifoundry.com/learn/atlas/l1-cache/) ### Related lessons - [On-chip interconnect fabric](https://semiaifoundry.com/learn/atlas/on-chip-network/) - [HBM controller and PHY](https://semiaifoundry.com/learn/atlas/hbm-controller/)