{"id":"kernel-locality-fusion","level":"software","name":"Kernel tiling, fusion and operand reuse","shortName":"Kernel tiling, fusion and operand reuse","role":"Keeps intermediate results close to their next consumer where resource limits allow.","description":"A kernel is a scheduled unit of device work, and tiling divides its operation into pieces that fit local memory and execution resources. Fusion places compatible operations in one kernel so intermediate values can remain in registers or shared memory instead of being written to HBM and read again. This can remove both traffic and launch overhead. The cost is a larger live working set, more register pressure and sometimes less freedom to use a tuned library kernel. OpenXLA documents a concrete GPU fusion path, but the general principle is to optimize data movement while preserving correctness and useful parallelism.","specs":[{"label":"Saved traffic","value":"Intermediate write plus later read","context":"Count bytes at the specific outer memory boundary."},{"label":"Local resource limit","value":"Registers and scratchpad per executing block","context":"Fusion can cause spills or reduce block residency."}],"interfaces":[{"name":"Input tensor tiles","direction":"in","signal":"Coalesced loads and reusable operands"},{"name":"Fused result","direction":"out","signal":"Final values after local intermediate computation"}],"tradeoff":"Fusion reduces intermediate traffic but may lose specialized library performance or exceed local storage; measure the combined kernel under representative shapes.","whyAI":"Elementwise chains, normalization and attention can spend more time moving intermediates than computing them, making locality central to throughput.","related":["kernel","gpu-occupancy-pipeline","attention-io-tiling"],"sources":[{"title":"XLA:GPU Architecture Overview","url":"https://openxla.org/xla/gpu_architecture"},{"title":"CUDA C++ Programming Guide","url":"https://docs.nvidia.com/cuda/cuda-c-programming-guide/"}],"scale":"Software abstraction layers mapped to physical memory and compute","kind":"workload","branch":"Model–machine co-design","subbranch":"Graph to executable operation","mechanism":["Identify intermediate tensors whose only consumers could run within the same device kernel.","Choose a tile that fits the required input, output and accumulator state in local resources.","Keep intermediate values in registers or scratchpad and execute consumers before evicting them.","Protect shared values with the synchronization required by the participating threads or groups.","Profile total time, spills and occupancy to verify that avoided outer-memory traffic outweighs added local pressure."],"misconception":"Fusing every operator is not always faster; resource pressure and library specialization can reverse the expected gain.","check":{"question":"What direct traffic saving comes from eliminating one N-byte temporary written and then read once?","options":["Up to 2N bytes at that memory boundary","Exactly N extra FLOPS","Unlimited memory capacity"],"answer":0,"explanation":"Removing the outer-memory write and subsequent read can avoid two transfers of the temporary, if it really remains local."},"science":[{"title":"Intermediate traffic saving","equation":"ΔQ = 2Ns","explanation":"N is temporary elements and s is bytes/element; ΔQ is avoided bytes for one write and one read.","assumptions":"The unfused version materializes both transfers and the fused version has no spill or other replacement traffic."}],"prerequisites":["compiler-layout-lowering","shared-sram"],"learningObjective":"Determine when fusion saves memory traffic and when local pressure can negate it.","workstream":"W11","visualFamily":"software-flow","engineeringExample":"Synthetic chain: an elementwise intermediate with one million two-byte values costs 2 MB to write and 2 MB to read. Fusion can avoid 4 MB, but a spill that recreates that traffic removes the expected advantage.","evidenceNotes":"The calculation is an upper-bound example. Runtime caches and optimized library behavior may reduce the unfused traffic, so actual savings require measurement.","labId":"engineering","scope":"Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.","deepLabId":"dataflow","investigationId":"execution","labScope":"Compare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations."}