The mechanism
A kernel is a scheduled unit of device work, and tiling divides its operation into pieces that fit local memory and execution resources. Fusion places compatible operations in one kernel so intermediate values can remain in registers or shared memory instead of being written to HBM and read again. This can remove both traffic and launch overhead. The cost is a larger live working set, more register pressure and sometimes less freedom to use a tuned library kernel. OpenXLA documents a concrete GPU fusion path, but the general principle is to optimize data movement while preserving correctness and useful parallelism.
- Identify intermediate tensors whose only consumers could run within the same device kernel.
- Choose a tile that fits the required input, output and accumulator state in local resources.
- Keep intermediate values in registers or scratchpad and execute consumers before evicting them.
- Protect shared values with the synchronization required by the participating threads or groups.
- Profile total time, spills and occupancy to verify that avoided outer-memory traffic outweighs added local pressure.
Science and assumptions
Intermediate traffic saving
Avoided intermediate traffic is twice element count times bytes per element: one eliminated write and one eliminated read.
Symbols and units
- ΔQ
- Traffic avoided by removing one materialized intermediate write and read
bytes
- N
- Intermediate element count
elements
- s
- Stored size per intermediate element
bytes/element
N is temporary elements and s is bytes/element; ΔQ is avoided bytes for one write and one read.
Assumptions The unfused version materializes both transfers and the fused version has no spill or other replacement traffic.
Engineering example
Synthetic chain: an elementwise intermediate with one million two-byte values costs 2 MB to write and 2 MB to read. Fusion can avoid 4 MB, but a spill that recreates that traffic removes the expected advantage.
Distinctions and tradeoffs
Fusing every operator is not always faster; resource pressure and library specialization can reverse the expected gain.
Fusion reduces intermediate traffic but may lose specialized library performance or exceed local storage; measure the combined kernel under representative shapes.
Elementwise chains, normalization and attention can spend more time moving intermediates than computing them, making locality central to throughput.
Specifications and interfaces
- Saved traffic
- Intermediate write plus later read
Count bytes at the specific outer memory boundary.
- Local resource limit
- Registers and scratchpad per executing block
Fusion can cause spills or reduce block residency.
Interfaces
- Input tensor tiles Input
- Coalesced loads and reusable operands
- Fused result Output
- Final values after local intermediate computation
Check your understanding
What direct traffic saving comes from eliminating one N-byte temporary written and then read once?
- Up to 2N bytes at that memory boundary
- Exactly N extra FLOPS
- Unlimited memory capacity
Answer and reasoning
Up to 2N bytes at that memory boundary
Removing the outer-memory write and subsequent read can avoid two transfers of the temporary, if it really remains local.
Evidence and scope
The calculation is an upper-bound example. Runtime caches and optimized library behavior may reduce the unfused traffic, so actual savings require measurement.
Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.