{"id":"streaming-multiprocessor","level":"die","name":"Streaming multiprocessor","shortName":"SM","role":"Schedules many threads and supplies the execution units and nearby storage that make GPU parallelism useful.","description":"An SM is a repeatable compute neighborhood inside a GPU. Instructions, thread state, arithmetic units, load/store units, registers, and local memories cooperate here. A program launches thread blocks; the GPU places eligible blocks on available SMs and divides their threads into warps. When one warp waits for data, another ready warp can make progress. This hides latency only when enough independent work and storage are available. A model layer does not correspond to one SM: its matrix tiles are distributed across many SMs, and the same hardware is reused for later layers. The atlas separates scheduling, execution, and storage to show why adding arithmetic alone cannot guarantee a faster model.","specs":[{"label":"Warp width","value":"32 threads","context":"CUDA execution grouping, not 32 entire model requests."},{"label":"Resident warps","value":"Up to 64 per SM","context":"Blackwell compute capability 10.0; occupancy also depends on resource use."},{"label":"Resident blocks","value":"Up to 32 per SM","context":"Blackwell 10.0 architectural limit, not a promised active count."}],"interfaces":[{"name":"Work dispatch","direction":"in","signal":"Kernel blocks and instructions"},{"name":"Operand path","direction":"in","signal":"Registers and staged tensor tiles"},{"name":"Results","direction":"out","signal":"Computed tiles and memory stores"}],"tradeoff":"More resident work can hide latency, but each block consumes finite registers and shared memory. Maximum occupancy can be slower than a carefully tiled kernel with fewer active blocks.","whyAI":"Transformer layers become thousands of tile operations scheduled across these neighborhoods. The SM is where software parallelism becomes simultaneous transistor activity.","related":["tensor-core","register-file","shared-sram","l1-cache","tensor-memory","gpu-die"],"sources":[{"title":"NVIDIA: Using CUDA Warp-Level Primitives","url":"https://developer.nvidia.com/blog/using-cuda-warp-level-primitives/"},{"title":"NVIDIA CUDA Blackwell Tuning Guide","url":"https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html"},{"title":"NVIDIA Nsight Compute: Roofline charts","url":"https://docs.nvidia.com/nsight-compute/ProfilingGuide/"},{"title":"NVIDIA CUDA Programming Guide: Introduction","url":"https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html"}],"scale":"Conceptual region of a centimeter-scale die; schematic floorplan","kind":"compute-neighborhood","mechanism":["A kernel launch creates a grid of thread blocks; the GPU places blocks on SMs where enough registers and shared memory are available.","Each block is divided into warps, and schedulers select ready instructions to feed arithmetic, tensor, or load/store pipelines.","Completed tiles are written or passed onward; the same SM then processes additional tiles and later model layers."],"misconception":"A model layer is not a dedicated physical layer of GPU silicon. Many SMs process pieces of one layer, then reuse the same circuits for the next.","check":{"question":"Why can a kernel with more arithmetic units still spend time waiting?","options":["It must wait for enough operands or independent ready work.","An SM can execute only one complete neural-network layer per second.","Registers permanently store all the model weights."],"answer":0,"explanation":"Arithmetic units are useful only when instructions and data are ready. Memory latency and resource limits can leave execution capacity idle."},"branch":"Compute architecture","subbranch":"Execution engines","science":[{"title":"Compute and data supply must balance","equation":"Pattainable ≤ min(Pcompute, Bmemory × I)","explanation":"P is throughput in operations per second, B is useful memory bandwidth in bytes per second, and I is arithmetic intensity in operations per byte moved at the chosen memory level. The sloped region is limited by operand delivery; the horizontal region is limited by arithmetic throughput. Tiling can raise I without changing the model by reusing a value several times after one load.","assumptions":"An upper-bound roofline model for a stated operation type and memory level. It excludes launch latency, dependency stalls, occupancy limits, and communication outside that level; peaks must use the same precision and sparsity convention."}],"history":[{"year":2006,"title":"CUDA opens GPU execution to general computation","significance":"NVIDIA introduced the CUDA architecture in 2006, allowing computational workloads to use GPU throughput without expressing the problem through graphics APIs. A programmable parallel hierarchy gave researchers a practical path from scientific and numerical code to large numbers of GPU execution units.","source":{"title":"NVIDIA CUDA Programming Guide: Introduction","url":"https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html"}}],"learningObjective":"Diagnose whether a kernel is compute or memory bound.","engineeringExample":"The roofline bound selects the lower of peak compute and bandwidth multiplied by arithmetic intensity; changing reuse can move the kernel between limits.","evidenceNotes":"Peak values and intensity must share precision, scope and time boundary.","visualFamily":"architecture-gpu","prerequisites":["tensor-core","register-file","clock"],"labId":"roofline","scope":"Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints.","investigationId":"execution","labScope":"Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints."}