{"id":"kernel","level":"software","name":"GPU kernel and thread hierarchy","shortName":"GPU kernel","role":"Maps an operation onto many coordinated logical threads or tiles.","description":"A GPU kernel is a unit of device code launched with an execution configuration. In CUDA's thread model, threads are grouped into blocks, and blocks form a grid. A block executes on a streaming multiprocessor and can use block-level shared memory and synchronization. The grid may contain far more blocks than the device can host simultaneously, so the logical grid is not a physical map of dedicated cores. The atlas animates blocks being assigned to available processors while preserving the distinction between work items and hardware. Efficient kernels arrange data access and reuse carefully. Arbitrary cross-block waiting is unsafe in a normal launch because blocks are not guaranteed to run in a particular order.","specs":[{"label":"Thread-model hierarchy","value":"Thread → block → grid","context":"Logical launch organization."},{"label":"Warp size","value":"32 CUDA threads","context":"Execution grouping in the documented programming model."},{"label":"Normal block ordering","value":"Not guaranteed","context":"Special supported cooperative mechanisms have additional rules."}],"interfaces":[{"name":"Input tensors and launch shape","direction":"in","signal":"Data pointers and parallel work configuration"},{"name":"Output tensors","direction":"out","signal":"Results written by the kernel"}],"tradeoff":"More parallel work can hide latency. Register use, shared memory, divergence, and data access constrain useful occupancy.","whyAI":"Kernels implement the actual tensor operations on GPUs. Tiling and reuse can matter as much as raw arithmetic capability.","related":["cuda","compiler","parallelism","register-file","feed-forward"],"sources":[{"title":"NVIDIA CUDA — Programming Model","url":"https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/programming-model.html"},{"title":"Berkeley Lab — Introduction to Roofline","url":"https://crd.lbl.gov/assets/Uploads/ECP19-Roofline-1-intro.pdf"}],"scale":"Logical work grid mapped over physical GPU resources","kind":"kernel","mechanism":["Divide the operation into indexed work items or tiles.","Launch blocks that the device assigns to available execution resources.","Reuse and synchronize data within supported boundaries, then write the result."],"misconception":"One launched thread is not one permanently dedicated physical GPU core; logical work is scheduled onto shared hardware.","check":{"question":"Can a normal kernel assume all grid blocks run simultaneously?","options":["Yes, every block has its own permanent core","Yes, if the grid is colored uniformly","No; block scheduling and order are not generally guaranteed"],"answer":2,"explanation":"A grid can exceed the device’s resident capacity, so ordinary cross-block dependencies must respect the programming model."},"branch":"Execution software","subbranch":"Kernel mapping","science":[{"title":"Roofline model","equation":"I = FLOPs / bytes moved; R ≤ min(P_peak, B_memory I)","explanation":"Arithmetic intensity connects the operation count to traffic at a selected memory boundary. Tiling can raise intensity by reusing fetched data, shifting a kernel toward its arithmetic ceiling.","assumptions":"Upper-bound model for a specified numeric format and memory level; measured performance is also limited by launch, occupancy, dependencies, and implementation details."}],"history":[],"prerequisites":["cuda","tensor-core","shared-sram"],"learningObjective":"Use arithmetic intensity to decide which roofline bound controls a specified kernel.","engineeringExample":"Synthetic kernel does 2 GFLOP and moves 1 GB at the selected memory boundary, so intensity is 2 FLOP/byte. A 10 TFLOP/s arithmetic peak and 1 TB/s memory rate yield min(10, 2) = 2 TFLOP/s; the optimistic execution lower bound is 1 ms.","evidenceNotes":"Numbers are synthetic and share a precision and byte boundary. The roofline is an upper bound on rate, not a measured runtime; dependencies, occupancy, launch overhead and cache behavior can reduce performance.","labId":"roofline","scope":"Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints.","investigationId":"execution","labScope":"Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints."}