# Occupancy and asynchronous tile pipelines

Canonical: https://semiaifoundry.com/learn/atlas/gpu-occupancy-pipeline/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Processing architectures · Atlas v1.3.0

# Occupancy and asynchronous tile pipelines

Keeps work ready while controlling registers, scratchpad capacity and synchronization.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#gpu-occupancy-pipeline)GPU execution group to package; schematic topology

Learning objectiveExplain the tension between occupancy, reuse and pipeline depth.

## The mechanism

A GPU can switch to another ready execution group while one waits for data. Occupancy measures how many groups are resident relative to a hardware limit; it does not measure how many arithmetic units are doing useful work. Each block reserves registers and shared memory, so a large tile can improve reuse while reducing the number of resident blocks. An asynchronous pipeline instead overlaps loading a future tile with computing the current tile. Hopper’s Tensor Memory Accelerator is a disclosed data-movement example. Correct buffer ownership and completion barriers matter as much as the number of threads launched.

1.  Calculate per-block register and scratchpad allocations before estimating how many blocks fit on a multiprocessor.
2.  Reserve separate buffers so a producer cannot overwrite a tile still being consumed.
3.  Start the next tile transfer while arithmetic uses the tile whose arrival has already been confirmed.
4.  Use completion and consumption barriers to prevent early reads and premature buffer reuse.
5.  Profile stalls and spills, then adjust tile shape or pipeline depth rather than maximizing occupancy alone.

## Science and assumptions

### Overlapped tile lower bound

$$
T_{\mathrm{tile}}\ge\max(T_{\mathrm{load}},T_{\mathrm{compute}})
$$

With ideal overlap, steady tile time is at least the larger of per-tile load time and compute time.

Symbols and units

T\_tile

Steady-state time per tile

s/tile

T\_load

Load resource time per tile

s/tile

T\_compute

Compute resource time per tile

s/tile

Both terms are seconds per tile. Independent engines can overlap their work after pipeline fill; the slower stage sets the steady-state bound.

Assumptions Enough buffers and work, independent resources and correct synchronization; shared bottlenecks can prevent full overlap.

## Engineering example

Synthetic pipeline: a tile takes 6 μs to load and 4 μs to compute. Serial execution needs 10 μs per tile; ideal overlap approaches 6 μs after startup, subject to buffer capacity and contention.

## Distinctions and tradeoffs

One hundred percent occupancy does not mean one hundred percent arithmetic utilization, and lower occupancy is not automatically a defect.

More buffering hides transfer time but uses shared memory; increasing occupancy can reduce per-block reuse or force register spills.

Attention and matrix kernels need a steady stream of operands; a launch with many threads can still leave the tensor units waiting.

## Specifications and interfaces

Occupancy

Resident warps / maximum resident warps

A resource-residency ratio, not FLOPS utilization.

Double buffering

Two live tile regions

Consumes additional scratchpad and needs explicit producer/consumer ordering.

### Interfaces

HBM or cache Input

Next operand tile copied asynchronously

Matrix engine Output

Ready tile and protected accumulator updates

## Check your understanding

### A larger tile lowers occupancy but reduces HBM reads. Which conclusion follows?

1.  Measure end-to-end kernel time before deciding
2.  The larger tile must be slower
3.  Occupancy determines all throughput

Answer and reasoning

Measure end-to-end kernel time before deciding

Less residency may be worthwhile if stronger reuse or overlap more than compensates for fewer available warps.

## Evidence and scope

TMA is a Hopper-specific example, while double buffering is a general method. No timing in this lesson is a vendor measurement.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [NVIDIA Hopper Tuning Guide](https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/gpu-occupancy-pipeline-76c0164cdd2b626d.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/)
-   [Register file](https://semiaifoundry.com/learn/atlas/register-file/)

### Related lessons

-   [Tensor Memory Accelerator](https://semiaifoundry.com/learn/atlas/tensor-memory-accelerator/)
-   [Kernel tiling, fusion and operand reuse](https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/)