{"id":"gpu-occupancy-pipeline","level":"compute","name":"Occupancy and asynchronous tile pipelines","shortName":"Occupancy and asynchronous tile pipelines","role":"Keeps work ready while controlling registers, scratchpad capacity and synchronization.","description":"A GPU can switch to another ready execution group while one waits for data. Occupancy measures how many groups are resident relative to a hardware limit; it does not measure how many arithmetic units are doing useful work. Each block reserves registers and shared memory, so a large tile can improve reuse while reducing the number of resident blocks. An asynchronous pipeline instead overlaps loading a future tile with computing the current tile. Hopper’s Tensor Memory Accelerator is a disclosed data-movement example. Correct buffer ownership and completion barriers matter as much as the number of threads launched.","specs":[{"label":"Occupancy","value":"Resident warps / maximum resident warps","context":"A resource-residency ratio, not FLOPS utilization."},{"label":"Double buffering","value":"Two live tile regions","context":"Consumes additional scratchpad and needs explicit producer/consumer ordering."}],"interfaces":[{"name":"HBM or cache","direction":"in","signal":"Next operand tile copied asynchronously"},{"name":"Matrix engine","direction":"out","signal":"Ready tile and protected accumulator updates"}],"tradeoff":"More buffering hides transfer time but uses shared memory; increasing occupancy can reduce per-block reuse or force register spills.","whyAI":"Attention and matrix kernels need a steady stream of operands; a launch with many threads can still leave the tensor units waiting.","related":["tensor-memory-accelerator","shared-sram","kernel-locality-fusion"],"sources":[{"title":"NVIDIA Hopper Tuning Guide","url":"https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html"}],"scale":"GPU execution group to package; schematic topology","kind":"architecture","branch":"Processing architectures","subbranch":"GPU execution and placement","mechanism":["Calculate per-block register and scratchpad allocations before estimating how many blocks fit on a multiprocessor.","Reserve separate buffers so a producer cannot overwrite a tile still being consumed.","Start the next tile transfer while arithmetic uses the tile whose arrival has already been confirmed.","Use completion and consumption barriers to prevent early reads and premature buffer reuse.","Profile stalls and spills, then adjust tile shape or pipeline depth rather than maximizing occupancy alone."],"misconception":"One hundred percent occupancy does not mean one hundred percent arithmetic utilization, and lower occupancy is not automatically a defect.","check":{"question":"A larger tile lowers occupancy but reduces HBM reads. Which conclusion follows?","options":["Measure end-to-end kernel time before deciding","The larger tile must be slower","Occupancy determines all throughput"],"answer":0,"explanation":"Less residency may be worthwhile if stronger reuse or overlap more than compensates for fewer available warps."},"science":[{"title":"Overlapped tile lower bound","equation":"Ttile ≥ max(Tload, Tcompute)","explanation":"Both terms are seconds per tile. Independent engines can overlap their work after pipeline fill; the slower stage sets the steady-state bound.","assumptions":"Enough buffers and work, independent resources and correct synchronization; shared bottlenecks can prevent full overlap."}],"prerequisites":["shared-sram","register-file"],"learningObjective":"Explain the tension between occupancy, reuse and pipeline depth.","workstream":"W07","visualFamily":"architecture-gpu","engineeringExample":"Synthetic pipeline: a tile takes 6 μs to load and 4 μs to compute. Serial execution needs 10 μs per tile; ideal overlap approaches 6 μs after startup, subject to buffer capacity and contention.","evidenceNotes":"TMA is a Hopper-specific example, while double buffering is a general method. No timing in this lesson is a vendor measurement.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}