The mechanism
An SM is a repeatable compute neighborhood inside a GPU. Instructions, thread state, arithmetic units, load/store units, registers, and local memories cooperate here. A program launches thread blocks; the GPU places eligible blocks on available SMs and divides their threads into warps. When one warp waits for data, another ready warp can make progress. This hides latency only when enough independent work and storage are available. A model layer does not correspond to one SM: its matrix tiles are distributed across many SMs, and the same hardware is reused for later layers. The atlas separates scheduling, execution, and storage to show why adding arithmetic alone cannot guarantee a faster model.
- A kernel launch creates a grid of thread blocks; the GPU places blocks on SMs where enough registers and shared memory are available.
- Each block is divided into warps, and schedulers select ready instructions to feed arithmetic, tensor, or load/store pipelines.
- Completed tiles are written or passed onward; the same SM then processes additional tiles and later model layers.
Science and assumptions
Compute and data supply must balance
Attainable throughput cannot exceed either arithmetic capacity or memory bandwidth times arithmetic intensity.
Symbols and units
- P_attainable
- Attainable arithmetic throughput
operations/s
- P_compute
- Arithmetic throughput ceiling for the selected format
operations/s
- B_memory
- Useful bandwidth at the selected memory boundary
bytes/s
- I
- Arithmetic intensity at that boundary
operations/byte
P is throughput in operations per second, B is useful memory bandwidth in bytes per second, and I is arithmetic intensity in operations per byte moved at the chosen memory level. The sloped region is limited by operand delivery; the horizontal region is limited by arithmetic throughput. Tiling can raise I without changing the model by reusing a value several times after one load.
Assumptions An upper-bound roofline model for a stated operation type and memory level. It excludes launch latency, dependency stalls, occupancy limits, and communication outside that level; peaks must use the same precision and sparsity convention.
Engineering example
The roofline bound selects the lower of peak compute and bandwidth multiplied by arithmetic intensity; changing reuse can move the kernel between limits.
Distinctions and tradeoffs
A model layer is not a dedicated physical layer of GPU silicon. Many SMs process pieces of one layer, then reuse the same circuits for the next.
More resident work can hide latency, but each block consumes finite registers and shared memory. Maximum occupancy can be slower than a carefully tiled kernel with fewer active blocks.
Transformer layers become thousands of tile operations scheduled across these neighborhoods. The SM is where software parallelism becomes simultaneous transistor activity.
Specifications and interfaces
- Warp width
- 32 threads
CUDA execution grouping, not 32 entire model requests.
- Resident warps
- Up to 64 per SM
Blackwell compute capability 10.0; occupancy also depends on resource use.
- Resident blocks
- Up to 32 per SM
Blackwell 10.0 architectural limit, not a promised active count.
Interfaces
- Work dispatch Input
- Kernel blocks and instructions
- Operand path Input
- Registers and staged tensor tiles
- Results Output
- Computed tiles and memory stores
Check your understanding
Why can a kernel with more arithmetic units still spend time waiting?
- It must wait for enough operands or independent ready work.
- An SM can execute only one complete neural-network layer per second.
- Registers permanently store all the model weights.
Answer and reasoning
It must wait for enough operands or independent ready work.
Arithmetic units are useful only when instructions and data are ready. Memory latency and resource limits can leave execution capacity idle.
Historical context
2006 · CUDA opens GPU execution to general computation
NVIDIA introduced the CUDA architecture in 2006, allowing computational workloads to use GPU throughput without expressing the problem through graphics APIs. A programmable parallel hierarchy gave researchers a practical path from scientific and numerical code to large numbers of GPU execution units.
Evidence and scope
Peak values and intensity must share precision, scope and time boundary.
Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints.