The mechanism
Shared memory is a software-managed scratchpad allocated to a thread block. Threads cooperate to load a tile, synchronize when required, and reuse it for several operations before replacing it. Its underlying storage is SRAM, which retains state while powered without the periodic refresh needed by DRAM. Shared memory is banked, so the access pattern matters: requests to different words in the same bank can serialize. Padding or rearranging a tile can remove those conflicts. The capacity is small compared with HBM, but a small tile can be reused many times. Requests that land on the same memory bank can serialize; changing a tile’s layout, for example by padding a matrix transpose, can reduce conflicts without changing its mathematical result.
- Cooperating threads or a copy engine bring a reusable tile from device memory into a block’s shared-memory allocation.
- A synchronization point establishes readiness; threads then read the tile repeatedly while computing several outputs.
- The block reuses the scratchpad for another tile, choosing an address layout that avoids conflicting accesses to the same bank.
Science and assumptions
Address layout determines bank conflicts
Divide the byte address by four bytes, take the floor, then take the remainder modulo 32 to obtain the bank index.
The explicit four-byte divisor makes the original address-unit convention visible; this mapping is configuration-specific.
Symbols and units
- a
- Byte-address offset within the mapped shared-memory address space
bytes
- bank(a)
- Bank index for the addressed word
integer 0–31
- 4 bytes
- Word width used by this mapping
bytes/word
- 32
- Number of shared-memory banks
count
a is a byte address for a 32-bit word in the documented shared-memory organization. Adjacent words land in different banks, while addresses separated by 128 bytes return to the same bank. A warp accessing different words in one bank may serialize. Padding a two-dimensional array changes its row stride and can transform a column access from conflicting to distributed.
Assumptions 32 banks and 4-byte word mapping in the referenced CUDA programming model. Wider instructions may be split into transactions; broadcast of the same read address is a special case and is not equivalent to different-word conflicts.
Engineering example
A tile that exceeds the available scratchpad must be split or reloaded, adding traffic that can erase arithmetic reuse.
Distinctions and tradeoffs
Shared memory and L1 cache are not interchangeable programming concepts. Shared memory is explicitly managed by software; cache retention is automatic.
Larger tiles improve reuse but consume more scratchpad and may reduce active blocks. Bank conflicts can waste this fast storage unless the layout matches the access pattern.
Attention kernels can avoid materializing large intermediate matrices in HBM by combining local tiles with online reductions. The savings come from a better dataflow, not extra model parameters.
Specifications and interfaces
- Address banks
- 32 banks
Successive 32-bit words map to successive banks in CUDA shared memory.
- Capacity per SM
- Up to 228 KB
Blackwell compute capability 10.0 shared-memory carveout.
- Per-block maximum
- 227 KB
One KB reserved by CUDA; dynamic opt-in required above static limits.
Interfaces
- Tile staging Input
- Coalesced loads or asynchronous copies
- Tile reuse Output
- Operands consumed by cooperating threads
- Synchronization Input
- Barriers establishing when staged data are ready
Check your understanding
Why might adding one padding column to a shared-memory tile improve performance?
- It increases HBM capacity.
- It can redirect accesses across different banks and avoid serialization.
- It adds more Tensor Cores to the SM.
Answer and reasoning
It can redirect accesses across different banks and avoid serialization.
Padding changes address-to-bank mapping. The arithmetic can stay identical while the memory requests stop competing for the same bank.
Evidence and scope
The bank and capacity behavior depends on a specific implementation.
Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.