# Streaming multiprocessor Canonical: https://semiaifoundry.com/learn/atlas/streaming-multiprocessor/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Compute architecture · Atlas v1.3.0 # Streaming multiprocessor Schedules many threads and supplies the execution units and nearby storage that make GPU parallelism useful. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#streaming-multiprocessor)Conceptual region of a centimeter-scale die; schematic floorplan Learning objectiveDiagnose whether a kernel is compute or memory bound. ## The mechanism An SM is a repeatable compute neighborhood inside a GPU. Instructions, thread state, arithmetic units, load/store units, registers, and local memories cooperate here. A program launches thread blocks; the GPU places eligible blocks on available SMs and divides their threads into warps. When one warp waits for data, another ready warp can make progress. This hides latency only when enough independent work and storage are available. A model layer does not correspond to one SM: its matrix tiles are distributed across many SMs, and the same hardware is reused for later layers. The atlas separates scheduling, execution, and storage to show why adding arithmetic alone cannot guarantee a faster model. 1. A kernel launch creates a grid of thread blocks; the GPU places blocks on SMs where enough registers and shared memory are available. 2. Each block is divided into warps, and schedulers select ready instructions to feed arithmetic, tensor, or load/store pipelines. 3. Completed tiles are written or passed onward; the same SM then processes additional tiles and later model layers. ## Science and assumptions ### Compute and data supply must balance $$ P_{\mathrm{attainable}}\le\min(P_{\mathrm{compute}},B_{\mathrm{memory}}I) $$ Attainable throughput cannot exceed either arithmetic capacity or memory bandwidth times arithmetic intensity. Symbols and units P\_attainable Attainable arithmetic throughput operations/s P\_compute Arithmetic throughput ceiling for the selected format operations/s B\_memory Useful bandwidth at the selected memory boundary bytes/s I Arithmetic intensity at that boundary operations/byte P is throughput in operations per second, B is useful memory bandwidth in bytes per second, and I is arithmetic intensity in operations per byte moved at the chosen memory level. The sloped region is limited by operand delivery; the horizontal region is limited by arithmetic throughput. Tiling can raise I without changing the model by reusing a value several times after one load. Assumptions An upper-bound roofline model for a stated operation type and memory level. It excludes launch latency, dependency stalls, occupancy limits, and communication outside that level; peaks must use the same precision and sparsity convention. ## Engineering example The roofline bound selects the lower of peak compute and bandwidth multiplied by arithmetic intensity; changing reuse can move the kernel between limits. ## Distinctions and tradeoffs A model layer is not a dedicated physical layer of GPU silicon. Many SMs process pieces of one layer, then reuse the same circuits for the next. More resident work can hide latency, but each block consumes finite registers and shared memory. Maximum occupancy can be slower than a carefully tiled kernel with fewer active blocks. Transformer layers become thousands of tile operations scheduled across these neighborhoods. The SM is where software parallelism becomes simultaneous transistor activity. ## Specifications and interfaces Warp width 32 threads CUDA execution grouping, not 32 entire model requests. Resident warps Up to 64 per SM Blackwell compute capability 10.0; occupancy also depends on resource use. Resident blocks Up to 32 per SM Blackwell 10.0 architectural limit, not a promised active count. ### Interfaces Work dispatch Input Kernel blocks and instructions Operand path Input Registers and staged tensor tiles Results Output Computed tiles and memory stores ## Check your understanding ### Why can a kernel with more arithmetic units still spend time waiting? 1. It must wait for enough operands or independent ready work. 2. An SM can execute only one complete neural-network layer per second. 3. Registers permanently store all the model weights. Answer and reasoning It must wait for enough operands or independent ready work. Arithmetic units are useful only when instructions and data are ready. Memory latency and resource limits can leave execution capacity idle. ## Historical context ### 2006 · CUDA opens GPU execution to general computation NVIDIA introduced the CUDA architecture in 2006, allowing computational workloads to use GPU throughput without expressing the problem through graphics APIs. A programmable parallel hierarchy gave researchers a practical path from scientific and numerical code to large numbers of GPU execution units. [NVIDIA CUDA Programming Guide: Introduction](https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html) ## Evidence and scope Peak values and intensity must share precision, scope and time boundary. Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints. ## Sources - [NVIDIA: Using CUDA Warp-Level Primitives](https://developer.nvidia.com/blog/using-cuda-warp-level-primitives/) - [NVIDIA CUDA Blackwell Tuning Guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html) - [NVIDIA Nsight Compute: Roofline charts](https://docs.nvidia.com/nsight-compute/ProfilingGuide/) - [NVIDIA CUDA Programming Guide: Introduction](https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/streaming-multiprocessor-edf22961f50ab64f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/) - [Register file](https://semiaifoundry.com/learn/atlas/register-file/) - [Clock tree and gated timing domains](https://semiaifoundry.com/learn/atlas/clock/) ### Related lessons - [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/) - [L1 and texture cache](https://semiaifoundry.com/learn/atlas/l1-cache/) - [Tensor Memory](https://semiaifoundry.com/learn/atlas/tensor-memory/) - [GPU logic dies](https://semiaifoundry.com/learn/atlas/gpu-die/)