The mechanism
A GPU kernel is a unit of device code launched with an execution configuration. In CUDA's thread model, threads are grouped into blocks, and blocks form a grid. A block executes on a streaming multiprocessor and can use block-level shared memory and synchronization. The grid may contain far more blocks than the device can host simultaneously, so the logical grid is not a physical map of dedicated cores. The atlas animates blocks being assigned to available processors while preserving the distinction between work items and hardware. Efficient kernels arrange data access and reuse carefully. Arbitrary cross-block waiting is unsafe in a normal launch because blocks are not guaranteed to run in a particular order.
- Divide the operation into indexed work items or tiles.
- Launch blocks that the device assigns to available execution resources.
- Reuse and synchronize data within supported boundaries, then write the result.
Science and assumptions
Roofline model
Arithmetic intensity is floating-point work divided by bytes moved. Throughput is bounded by the smaller of arithmetic peak and memory bandwidth times intensity.
Symbols and units
- I
- Arithmetic intensity at the selected memory boundary
FLOP/byte
- F
- Work in the declared floating-point convention
FLOP
- M
- Bytes moved at that boundary
bytes
- R
- Attainable arithmetic throughput
FLOP/s
- P_peak
- Compatible arithmetic peak
FLOP/s
- B_memory
- Useful memory bandwidth
bytes/s
Arithmetic intensity connects the operation count to traffic at a selected memory boundary. Tiling can raise intensity by reusing fetched data, shifting a kernel toward its arithmetic ceiling.
Assumptions Upper-bound model for a specified numeric format and memory level; measured performance is also limited by launch, occupancy, dependencies, and implementation details.
Engineering example
Synthetic kernel does 2 GFLOP and moves 1 GB at the selected memory boundary, so intensity is 2 FLOP/byte. A 10 TFLOP/s arithmetic peak and 1 TB/s memory rate yield min(10, 2) = 2 TFLOP/s; the optimistic execution lower bound is 1 ms.
Distinctions and tradeoffs
One launched thread is not one permanently dedicated physical GPU core; logical work is scheduled onto shared hardware.
More parallel work can hide latency. Register use, shared memory, divergence, and data access constrain useful occupancy.
Kernels implement the actual tensor operations on GPUs. Tiling and reuse can matter as much as raw arithmetic capability.
Specifications and interfaces
- Thread-model hierarchy
- Thread → block → grid
Logical launch organization.
- Warp size
- 32 CUDA threads
Execution grouping in the documented programming model.
- Normal block ordering
- Not guaranteed
Special supported cooperative mechanisms have additional rules.
Interfaces
- Input tensors and launch shape Input
- Data pointers and parallel work configuration
- Output tensors Output
- Results written by the kernel
Check your understanding
Can a normal kernel assume all grid blocks run simultaneously?
- Yes, every block has its own permanent core
- Yes, if the grid is colored uniformly
- No; block scheduling and order are not generally guaranteed
Answer and reasoning
No; block scheduling and order are not generally guaranteed
A grid can exceed the device’s resident capacity, so ordinary cross-block dependencies must respect the programming model.
Evidence and scope
Numbers are synthetic and share a precision and byte boundary. The roofline is an upper bound on rate, not a measured runtime; dependencies, occupancy, launch overhead and cache behavior can reduce performance.
Compare arithmetic and memory-bandwidth ceilings; occupancy and instruction dependencies remain omitted constraints.