{
  "schemaVersion": 1,
  "id": "atlas-lesson-streaming-multiprocessor",
  "title": "Streaming multiprocessor",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Schedules many threads and supplies the execution units and nearby storage that make GPU parallelism useful.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/streaming-multiprocessor/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Compute architecture",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Peak values and intensity must share precision, scope and time boundary.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/streaming-multiprocessor-edf22961f50ab64f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#streaming-multiprocessor",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "927b1c7aa3264c804716d7a413f1f1dc72c812b8948ea723aa5be776597b938d",
  "contentLength": 6441,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-streaming-multiprocessor.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-streaming-multiprocessor.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Streaming multiprocessor",
      "start": 37,
      "end": 394
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 394,
      "end": 1506
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1506,
      "end": 1534
    },
    {
      "id": "section-4",
      "title": "Compute and data supply must balance",
      "start": 1534,
      "end": 2696
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2696,
      "end": 2875
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2875,
      "end": 3411
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3411,
      "end": 3734
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3734,
      "end": 3908
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3908,
      "end": 3937
    },
    {
      "id": "section-10",
      "title": "Why can a kernel with more arithmetic units still spend time waiting?",
      "start": 3937,
      "end": 4430
    },
    {
      "id": "section-11",
      "title": "Historical context",
      "start": 4430,
      "end": 4453
    },
    {
      "id": "section-12",
      "title": "2006 · CUDA opens GPU execution to general computation",
      "start": 4453,
      "end": 4950
    },
    {
      "id": "section-13",
      "title": "Evidence and scope",
      "start": 4950,
      "end": 5165
    },
    {
      "id": "section-14",
      "title": "Sources",
      "start": 5165,
      "end": 5840
    },
    {
      "id": "section-15",
      "title": "Continue exploring",
      "start": 5840,
      "end": 5863
    },
    {
      "id": "section-16",
      "title": "Foundations",
      "start": 5863,
      "end": 6125
    },
    {
      "id": "section-17",
      "title": "Related lessons",
      "start": 6125,
      "end": 6441
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#streaming-multiprocessor"
    },
    {
      "title": "NVIDIA CUDA Programming Guide: Introduction",
      "url": "https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html"
    },
    {
      "title": "NVIDIA: Using CUDA Warp-Level Primitives",
      "url": "https://developer.nvidia.com/blog/using-cuda-warp-level-primitives/"
    },
    {
      "title": "NVIDIA CUDA Blackwell Tuning Guide",
      "url": "https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html"
    },
    {
      "title": "NVIDIA Nsight Compute: Roofline charts",
      "url": "https://docs.nvidia.com/nsight-compute/ProfilingGuide/"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/streaming-multiprocessor-edf22961f50ab64f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Tensor Core matrix unit",
      "url": "https://semiaifoundry.com/learn/atlas/tensor-core/"
    },
    {
      "title": "Register file",
      "url": "https://semiaifoundry.com/learn/atlas/register-file/"
    },
    {
      "title": "Clock tree and gated timing domains",
      "url": "https://semiaifoundry.com/learn/atlas/clock/"
    },
    {
      "title": "Shared memory SRAM",
      "url": "https://semiaifoundry.com/learn/atlas/shared-sram/"
    },
    {
      "title": "L1 and texture cache",
      "url": "https://semiaifoundry.com/learn/atlas/l1-cache/"
    },
    {
      "title": "Tensor Memory",
      "url": "https://semiaifoundry.com/learn/atlas/tensor-memory/"
    },
    {
      "title": "GPU logic dies",
      "url": "https://semiaifoundry.com/learn/atlas/gpu-die/"
    }
  ],
  "recordHash": "bf80cb7861f24b5a67b46c01256835c5fc2dad0c3f5a90ae530155a6460a9d19",
  "contentFormat": "text/markdown",
  "content": "Compute architecture · Atlas v1.3.0\n\n# Streaming multiprocessor\n\nSchedules many threads and supplies the execution units and nearby storage that make GPU parallelism useful.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#streaming-multiprocessor)Conceptual region of a centimeter-scale die; schematic floorplan\n\nLearning objectiveDiagnose whether a kernel is compute or memory bound.\n\n## The mechanism\n\nAn SM is a repeatable compute neighborhood inside a GPU. Instructions, thread state, arithmetic units, load/store units, registers, and local memories cooperate here. A program launches thread blocks; the GPU places eligible blocks on available SMs and divides their threads into warps. When one warp waits for data, another ready warp can make progress. This hides latency only when enough independent work and storage are available. A model layer does not correspond to one SM: its matrix tiles are distributed across many SMs, and the same hardware is reused for later layers. The atlas separates scheduling, execution, and storage to show why adding arithmetic alone cannot guarantee a faster model.\n\n1.  A kernel launch creates a grid of thread blocks; the GPU places blocks on SMs where enough registers and shared memory are available.\n2.  Each block is divided into warps, and schedulers select ready instructions to feed arithmetic, tensor, or load/store pipelines.\n3.  Completed tiles are written or passed onward; the same SM then processes additional tiles and later model layers.\n\n## Science and assumptions\n\n### Compute and data supply must balance\n\n$$\nP_{\\mathrm{attainable}}\\le\\min(P_{\\mathrm{compute}},B_{\\mathrm{memory}}I)\n$$\n\nAttainable throughput cannot exceed either arithmetic capacity or memory bandwidth times arithmetic intensity.\n\nSymbols and units\n\nP\\_attainable\n\nAttainable arithmetic throughput\n\noperations/s\n\nP\\_compute\n\nArithmetic throughput ceiling for the selected format\n\noperations/s\n\nB\\_memory\n\nUseful bandwidth at the selected memory boundary\n\nbytes/s\n\nI\n\nArithmetic intensity at that boundary\n\noperations/byte\n\nP is throughput in operations per second, B is useful memory bandwidth in bytes per second, and I is arithmetic intensity in operations per byte moved at the chosen memory level. The sloped region is limited by operand delivery; the horizontal region is limited by arithmetic throughput. Tiling can raise I without changing the model by reusing a value several times after one load.\n\nAssumptions An upper-bound roofline model for a stated operation type and memory level. It excludes launch latency, dependency stalls, occupancy limits, and communication outside that level; peaks must use the same precision and sparsity convention.\n\n## Engineering example\n\nThe roofline bound selects the lower of peak compute and bandwidth multiplied by arithmetic intensity; changing reuse can move the kernel between limits.\n\n## Distinctions and tradeoffs\n\nA model layer is not a dedicated physical layer of GPU silicon. Many SMs process pieces of one layer, then reuse the same circuits for the next.\n\nMore resident work can hide latency, but each block consumes finite registers and shared memory. Maximum occupancy can be slower than a carefully tiled kernel with fewer active blocks.\n\nTransformer layers become thousands of tile operations scheduled across these neighborhoods. The SM is where software parallelism becomes simultaneous transistor activity.\n\n## Specifications and interfaces\n\nWarp width\n\n32 threads\n\nCUDA execution grouping, not 32 entire model requests.\n\nResident warps\n\nUp to 64 per SM\n\nBlackwell compute capability 10.0; occupancy also depends on resource use.\n\nResident blocks\n\nUp to 32 per SM\n\nBlackwell 10.0 architectural limit, not a promised active count.\n\n### Interfaces\n\nWork dispatch Input\n\nKernel blocks and instructions\n\nOperand path Input\n\nRegisters and staged tensor tiles\n\nResults Output\n\nComputed tiles and memory stores\n\n## Check your understanding\n\n### Why can a kernel with more arithmetic units still spend time waiting?\n\n1.  It must wait for enough operands or independent ready work.\n2.  An SM can execute only one complete neural-network layer per second.\n3.  Registers permanently store all the model weights.\n\nAnswer and reasoning\n\nIt must wait for enough operands or independent ready work.\n\nArithmetic units are useful only when instructions and data are ready. Memory latency and resource limits can leave execution capacity idle.\n\n## Historical context\n\n### 2006 · CUDA opens GPU execution to general computation\n\nNVIDIA introduced the CUDA architecture in 2006, allowing computational workloads to use GPU throughput without expressing the problem through graphics APIs. A programmable parallel hierarchy gave researchers a practical path from scientific and numerical code to large numbers of GPU execution units.\n\n[NVIDIA CUDA Programming Guide: Introduction](https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html)\n\n## Evidence and scope\n\nPeak values and intensity must share precision, scope and time boundary.\n\nCompare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints.\n\n## Sources\n\n-   [NVIDIA: Using CUDA Warp-Level Primitives](https://developer.nvidia.com/blog/using-cuda-warp-level-primitives/)\n-   [NVIDIA CUDA Blackwell Tuning Guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html)\n-   [NVIDIA Nsight Compute: Roofline charts](https://docs.nvidia.com/nsight-compute/ProfilingGuide/)\n-   [NVIDIA CUDA Programming Guide: Introduction](https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/introduction.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/streaming-multiprocessor-edf22961f50ab64f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/)\n-   [Register file](https://semiaifoundry.com/learn/atlas/register-file/)\n-   [Clock tree and gated timing domains](https://semiaifoundry.com/learn/atlas/clock/)\n\n### Related lessons\n\n-   [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/)\n-   [L1 and texture cache](https://semiaifoundry.com/learn/atlas/l1-cache/)\n-   [Tensor Memory](https://semiaifoundry.com/learn/atlas/tensor-memory/)\n-   [GPU logic dies](https://semiaifoundry.com/learn/atlas/gpu-die/)"
}