{
  "schemaVersion": 1,
  "id": "atlas-lesson-kernel-locality-fusion",
  "title": "Kernel tiling, fusion and operand reuse",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Keeps intermediate results close to their next consumer where resource limits allow.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The calculation is an upper-bound example. Runtime caches and optimized library behavior may reduce the unfused traffic, so actual savings require measurement.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/kernel-locality-fusion-5ebaefca6a873058.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#kernel-locality-fusion",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "878b3639b9b3829d382348ac489eb8287e7f45109feff75b6c9da6cd688780ef",
  "contentLength": 5036,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-kernel-locality-fusion.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-kernel-locality-fusion.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Kernel tiling, fusion and operand reuse",
      "start": 40,
      "end": 415
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 415,
      "end": 1572
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1572,
      "end": 1600
    },
    {
      "id": "section-4",
      "title": "Intermediate traffic saving",
      "start": 1600,
      "end": 2201
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2201,
      "end": 2437
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2437,
      "end": 2903
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2903,
      "end": 3160
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3160,
      "end": 3313
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3313,
      "end": 3342
    },
    {
      "id": "section-10",
      "title": "What direct traffic saving comes from eliminating one N-byte temporary written and then read once?",
      "start": 3342,
      "end": 3731
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3731,
      "end": 4100
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4100,
      "end": 4481
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4481,
      "end": 4504
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4504,
      "end": 4716
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4716,
      "end": 5036
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#kernel-locality-fusion"
    },
    {
      "title": "XLA:GPU Architecture Overview",
      "url": "https://openxla.org/xla/gpu_architecture"
    },
    {
      "title": "CUDA C++ Programming Guide",
      "url": "https://docs.nvidia.com/cuda/cuda-c-programming-guide/"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/kernel-locality-fusion-5ebaefca6a873058.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Compiler layout, lowering and buffer planning",
      "url": "https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/"
    },
    {
      "title": "Shared memory SRAM",
      "url": "https://semiaifoundry.com/learn/atlas/shared-sram/"
    },
    {
      "title": "GPU kernel and thread hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/kernel/"
    },
    {
      "title": "Occupancy and asynchronous tile pipelines",
      "url": "https://semiaifoundry.com/learn/atlas/gpu-occupancy-pipeline/"
    },
    {
      "title": "IO-aware attention and online softmax",
      "url": "https://semiaifoundry.com/learn/atlas/attention-io-tiling/"
    }
  ],
  "recordHash": "d39496cfd03d5f6bf7876bcc0ffca9f082eb4871d33bc484f46f896ad895bb82",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# Kernel tiling, fusion and operand reuse\n\nKeeps intermediate results close to their next consumer where resource limits allow.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#kernel-locality-fusion)Software abstraction layers mapped to physical memory and compute\n\nLearning objectiveDetermine when fusion saves memory traffic and when local pressure can negate it.\n\n## The mechanism\n\nA kernel is a scheduled unit of device work, and tiling divides its operation into pieces that fit local memory and execution resources. Fusion places compatible operations in one kernel so intermediate values can remain in registers or shared memory instead of being written to HBM and read again. This can remove both traffic and launch overhead. The cost is a larger live working set, more register pressure and sometimes less freedom to use a tuned library kernel. OpenXLA documents a concrete GPU fusion path, but the general principle is to optimize data movement while preserving correctness and useful parallelism.\n\n1.  Identify intermediate tensors whose only consumers could run within the same device kernel.\n2.  Choose a tile that fits the required input, output and accumulator state in local resources.\n3.  Keep intermediate values in registers or scratchpad and execute consumers before evicting them.\n4.  Protect shared values with the synchronization required by the participating threads or groups.\n5.  Profile total time, spills and occupancy to verify that avoided outer-memory traffic outweighs added local pressure.\n\n## Science and assumptions\n\n### Intermediate traffic saving\n\n$$\n\\Delta Q=2Ns\n$$\n\nAvoided intermediate traffic is twice element count times bytes per element: one eliminated write and one eliminated read.\n\nSymbols and units\n\nΔQ\n\nTraffic avoided by removing one materialized intermediate write and read\n\nbytes\n\nN\n\nIntermediate element count\n\nelements\n\ns\n\nStored size per intermediate element\n\nbytes/element\n\nN is temporary elements and s is bytes/element; ΔQ is avoided bytes for one write and one read.\n\nAssumptions The unfused version materializes both transfers and the fused version has no spill or other replacement traffic.\n\n## Engineering example\n\nSynthetic chain: an elementwise intermediate with one million two-byte values costs 2 MB to write and 2 MB to read. Fusion can avoid 4 MB, but a spill that recreates that traffic removes the expected advantage.\n\n## Distinctions and tradeoffs\n\nFusing every operator is not always faster; resource pressure and library specialization can reverse the expected gain.\n\nFusion reduces intermediate traffic but may lose specialized library performance or exceed local storage; measure the combined kernel under representative shapes.\n\nElementwise chains, normalization and attention can spend more time moving intermediates than computing them, making locality central to throughput.\n\n## Specifications and interfaces\n\nSaved traffic\n\nIntermediate write plus later read\n\nCount bytes at the specific outer memory boundary.\n\nLocal resource limit\n\nRegisters and scratchpad per executing block\n\nFusion can cause spills or reduce block residency.\n\n### Interfaces\n\nInput tensor tiles Input\n\nCoalesced loads and reusable operands\n\nFused result Output\n\nFinal values after local intermediate computation\n\n## Check your understanding\n\n### What direct traffic saving comes from eliminating one N-byte temporary written and then read once?\n\n1.  Up to 2N bytes at that memory boundary\n2.  Exactly N extra FLOPS\n3.  Unlimited memory capacity\n\nAnswer and reasoning\n\nUp to 2N bytes at that memory boundary\n\nRemoving the outer-memory write and subsequent read can avoid two transfers of the temporary, if it really remains local.\n\n## Evidence and scope\n\nThe calculation is an upper-bound example. Runtime caches and optimized library behavior may reduce the unfused traffic, so actual savings require measurement.\n\nCompare unblocked and tiled matrix traffic for fixed arithmetic; architecture-specific instructions, fusion and attention algorithms are context rather than simulated implementations.\n\n## Sources\n\n-   [XLA:GPU Architecture Overview](https://openxla.org/xla/gpu_architecture)\n-   [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/kernel-locality-fusion-5ebaefca6a873058.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Compiler layout, lowering and buffer planning](https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/)\n-   [Shared memory SRAM](https://semiaifoundry.com/learn/atlas/shared-sram/)\n\n### Related lessons\n\n-   [GPU kernel and thread hierarchy](https://semiaifoundry.com/learn/atlas/kernel/)\n-   [Occupancy and asynchronous tile pipelines](https://semiaifoundry.com/learn/atlas/gpu-occupancy-pipeline/)\n-   [IO-aware attention and online softmax](https://semiaifoundry.com/learn/atlas/attention-io-tiling/)"
}