Occupancy and asynchronous tile pipelines

The mechanism

A GPU can switch to another ready execution group while one waits for data. Occupancy measures how many groups are resident relative to a hardware limit; it does not measure how many arithmetic units are doing useful work. Each block reserves registers and shared memory, so a large tile can improve reuse while reducing the number of resident blocks. An asynchronous pipeline instead overlaps loading a future tile with computing the current tile. Hopper’s Tensor Memory Accelerator is a disclosed data-movement example. Correct buffer ownership and completion barriers matter as much as the number of threads launched.

Calculate per-block register and scratchpad allocations before estimating how many blocks fit on a multiprocessor.
Reserve separate buffers so a producer cannot overwrite a tile still being consumed.
Start the next tile transfer while arithmetic uses the tile whose arrival has already been confirmed.
Use completion and consumption barriers to prevent early reads and premature buffer reuse.
Profile stalls and spills, then adjust tile shape or pipeline depth rather than maximizing occupancy alone.

Science and assumptions

Overlapped tile lower bound

T_{\mathrm{tile}}\ge\max(T_{\mathrm{load}},T_{\mathrm{compute}})

With ideal overlap, steady tile time is at least the larger of per-tile load time and compute time.

Symbols and units

T_tile: Steady-state time per tile
s/tile
T_load: Load resource time per tile
s/tile
T_compute: Compute resource time per tile
s/tile

Both terms are seconds per tile. Independent engines can overlap their work after pipeline fill; the slower stage sets the steady-state bound.

Assumptions Enough buffers and work, independent resources and correct synchronization; shared bottlenecks can prevent full overlap.

Engineering example

Synthetic pipeline: a tile takes 6 μs to load and 4 μs to compute. Serial execution needs 10 μs per tile; ideal overlap approaches 6 μs after startup, subject to buffer capacity and contention.

Distinctions and tradeoffs

One hundred percent occupancy does not mean one hundred percent arithmetic utilization, and lower occupancy is not automatically a defect.

More buffering hides transfer time but uses shared memory; increasing occupancy can reduce per-block reuse or force register spills.

Attention and matrix kernels need a steady stream of operands; a launch with many threads can still leave the tensor units waiting.

Specifications and interfaces

Occupancy: Resident warps / maximum resident warps
A resource-residency ratio, not FLOPS utilization.
Double buffering: Two live tile regions
Consumes additional scratchpad and needs explicit producer/consumer ordering.

Interfaces

HBM or cache Input: Next operand tile copied asynchronously
Matrix engine Output: Ready tile and protected accumulator updates

Check your understanding

A larger tile lowers occupancy but reduces HBM reads. Which conclusion follows?

Measure end-to-end kernel time before deciding
The larger tile must be slower
Occupancy determines all throughput

Answer and reasoning

Measure end-to-end kernel time before deciding

Less residency may be worthwhile if stronger reuse or overlap more than compensates for fewer available warps.

Evidence and scope

TMA is a Hopper-specific example, while double buffering is a general method. No timing in this lesson is a vendor measurement.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.