L1 and texture cache

The mechanism

The L1 cache is the first automatic cache close to an SM’s execution units. When threads request memory, hardware checks whether useful data are already nearby. Hits can avoid a longer journey; misses continue toward lower levels of the hierarchy. Consecutive thread addresses can be coalesced into fewer memory transactions, while scattered addresses may move many unused bytes. Blackwell shares a physical capacity budget among L1, texture functionality, and the shared-memory carveout. This differs from the explicit ownership and lifetime of software-managed shared memory. A cache hit serves a load locally, while a miss requests data farther down the hierarchy. Reshaping a warp’s addresses can change transferred bytes even when it computes the same values.

Threads issue addresses, and the memory system combines compatible nearby requests into transactions.
A cache lookup supplies data that are already present near the SM; a miss continues toward the next memory level.
Returned data may populate the cache, making later local or nearby accesses less expensive.

Science and assumptions

Useful bytes versus bytes actually moved

\eta_{\mathrm{useful}}=\frac{M_{\mathrm{useful}}}{M_{\mathrm{transferred}}},\qquad B_{\mathrm{useful}}=\eta_{\mathrm{useful}}B_{\mathrm{transferred}}

Useful-byte efficiency is useful bytes divided by transferred bytes. Multiplying transferred bandwidth by that efficiency gives useful bandwidth.

Symbols and units

η_useful: Useful-byte fraction for the declared traffic accounting
fraction
M_useful: Useful bytes in the counted transfers
bytes
M_transferred: All bytes physically moved for those transfers
bytes
B_useful: Useful byte rate
bytes/s
B_transferred: Transferred byte rate over the same interval
bytes/s

Museful counts bytes the computation needs; Mtransferred counts all bytes fetched in the memory transactions serving those needs. A scattered access pattern can move many unused bytes, reducing effective bandwidth even if the physical interface is busy. Coalescing neighboring thread requests and reusing cache lines improve the fraction of movement that performs useful work.

Assumptions Accounting over the same interval and memory boundary. Count unique or repeated demand consistently, and treat compression, cache hits, write traffic, and replayed transactions separately when profiling.

Engineering example

If only half of transferred cache-line bytes feed useful arithmetic, η_useful is 0.5 even though the interface moved every byte.

Distinctions and tradeoffs

A 256 KB combined L1/texture/shared-memory capacity does not mean 256 KB of L1 plus another 256 KB of shared memory. They share a capacity budget.

A cache helps when nearby or recent data will be reused, but unpredictable access can defeat locality. Increasing software-managed shared memory can improve explicit reuse while reducing space available to automatic caching.

Embedding lookups, normalization, and intermediate tensor access can become memory limited. Data layout determines whether the hierarchy moves useful numbers or mostly unused bytes.

Specifications and interfaces

Combined capacity: 256 KB per SM
Blackwell B200 L1/texture/shared-memory aggregate; not 256 KB of L1 plus shared memory.
Allocation: Configurable carveout
A larger shared-memory reservation leaves a different cache allocation.
Coalescing example: 32-byte transaction segments
CUDA programming guidance; actual request handling depends on instruction and access pattern.

Interfaces

Memory request Input: Warp load/store addresses
Cache hit Output: Locally available data
Cache miss Output: Request toward L2 and device memory

Check your understanding

Which access pattern generally makes better use of memory transactions?

Nearby threads requesting neighboring useful values.
Every thread requesting an unrelated distant address.
Every load deliberately bypassing all reuse.

Answer and reasoning

Nearby threads requesting neighboring useful values.

Coalescing can serve neighboring addresses with fewer transactions, reducing the number of unused bytes moved for the same computation.

Evidence and scope

The ratio is workload- and access-pattern-specific.

Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.