The mechanism
L2 sits farther from individual execution units and serves requests across the GPU. A tile loaded by one part of the chip can remain available for later work, reducing traffic to external HBM. The cache is physically distributed even when software sees one device address space. Address placement and work placement can therefore influence the distance data travels. Persistence controls can help protect selected reuse, but they do not make the entire model fit into cache. A useful view shows multiple SMs converging on cache slices, with a counter for bytes served locally versus bytes fetched from HBM. Cache capacity should be compared with a working set, not confused with total model storage.
- Requests that cannot be satisfied close to an SM reach shared L2 cache slices through the internal fabric.
- An L2 hit returns retained data; a miss asks the memory controllers to fetch data from HBM.
- Reuse by later work can avoid additional HBM trips, while replacement policies and working-set size determine what remains available.
Science and assumptions
Average access cost across two cache levels
Average access time equals L1 lookup time plus the L1 miss fraction times L2 lookup time and its conditional HBM miss penalty.
Symbols and units
- AMAT
- Average memory access time in the sequential model
s
- t_L1
- L1 lookup time
s
- m_L1
- Fraction of accesses missing L1
fraction
- t_L2
- Additional L2 lookup time after an L1 miss
s
- m_L2
- L2 miss fraction conditional on an L1 miss
fraction
- t_HBM
- Additional service time after an L2 miss
s
tL1 and tL2 are lookup times; mL1 is the fraction missing in L1 and mL2 is the fraction of those requests that also miss in L2. tHBM is the additional service time after an L2 miss. Lowering mL2 prevents some expensive trips to HBM, making a cache useful even when it stores only a small portion of total device memory.
Assumptions A sequential lookup model with conditional miss rates and incremental latency terms. Real GPUs overlap requests, queue transactions, and hide latency with other work, so AMAT is not the same as kernel execution time.
Engineering example
A 5% L1 miss rate makes the L2 path visible; a 20% L2 miss rate then exposes the HBM term in the nested expression.
Distinctions and tradeoffs
A unified device address space does not mean every physical memory route has identical cost. Distributed cache and internal locality can still matter.
A larger cache can save external bandwidth but consumes expensive logic-die area. Locality and scheduling can matter as much as nominal capacity when requests cross internal partitions.
Reused weights and intermediate tensors can be served without another HBM access. This helps only when the active working set and access pattern fit the cache hierarchy.
Specifications and interfaces
- Published capacity
- 126 MB
NVIDIA Blackwell Tuning Guide labels this as GB200 GPU L2 capacity.
- Scope
- Shared device cache
Physically distributed; not private storage for each SM.
- Reuse control
- L2 persistence hints
CUDA policies influence retention; they are not capacity guarantees.
Interfaces
- SM requests Input
- Loads, stores, and atomic operations
- Cache response Output
- Reused data to requesting compute units
- Miss/refill Output
- Transactions to HBM memory controllers
Check your understanding
When does a larger L2 cache most directly help?
- Whenever a model has more vocabulary entries, regardless of access.
- Only when the network cable is unplugged.
- When useful data are reused before they are evicted.
Answer and reasoning
When useful data are reused before they are evicted.
A cache reduces external traffic when it retains data that will be needed again. Capacity alone does not create reuse.
Evidence and scope
AMAT is an average model and does not capture overlap or queue contention.
Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.