{
  "schemaVersion": 1,
  "id": "atlas-lesson-attention-io-tiling",
  "title": "IO-aware attention and online softmax",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Computes attention while reducing large intermediate transfers to outer memory.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/attention-io-tiling/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The example is a storage calculation, not a promised speedup. Exactness refers to the algorithm’s attention definition; finite-precision evaluation still needs testing.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/attention-io-tiling-430677d2c8cf857a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#attention-io-tiling",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "249ed34b9273038259b4fc21799b57c19a0921955146e33690d12eee2ca1f8ea",
  "contentLength": 4936,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-attention-io-tiling.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-attention-io-tiling.md",
  "sections": [
    {
      "id": "section-1",
      "title": "IO-aware attention and online softmax",
      "start": 40,
      "end": 412
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 412,
      "end": 1687
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1687,
      "end": 1715
    },
    {
      "id": "section-4",
      "title": "Score storage",
      "start": 1715,
      "end": 2327
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2327,
      "end": 2580
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2580,
      "end": 3050
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3050,
      "end": 3320
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3320,
      "end": 3497
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3497,
      "end": 3526
    },
    {
      "id": "section-10",
      "title": "What is the main avoided object in tiled IO-aware attention?",
      "start": 3526,
      "end": 3920
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3920,
      "end": 4257
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4257,
      "end": 4584
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4584,
      "end": 4607
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4607,
      "end": 4805
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4805,
      "end": 4936
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#attention-io-tiling"
    },
    {
      "title": "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness",
      "url": "https://arxiv.org/abs/2205.14135"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/attention-io-tiling-430677d2c8cf857a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Kernel tiling, fusion and operand reuse",
      "url": "https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/"
    },
    {
      "title": "Occupancy and asynchronous tile pipelines",
      "url": "https://semiaifoundry.com/learn/atlas/gpu-occupancy-pipeline/"
    }
  ],
  "recordHash": "3ebb517fa63e684df53b554119161f77a325f5f8d526c1b0bf9337a6f0194b0e",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# IO-aware attention and online softmax\n\nComputes attention while reducing large intermediate transfers to outer memory.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#attention-io-tiling)One model step to accelerator memory hierarchy\n\nLearning objectiveExplain how tiling and online normalization reduce attention IO without confusing exact and sparse methods.\n\n## The mechanism\n\nStandard attention can create a score matrix with one entry for each query–key pair. Materializing that matrix in HBM becomes expensive as sequence length grows. FlashAttention reorganizes the calculation into tiles, keeping useful intermediates in faster local memory and maintaining the normalization information needed to combine blocks. The algorithm still computes exact attention in its stated numerical setting; it is different from deleting attention connections or approximating the model. Backward computation can recompute some intermediates rather than storing them all. The benefit depends on the memory hierarchy, tile shape, sequence length and implementation, not simply on changing the name of the attention kernel.\n\n1.  Partition query, key and value matrices into tiles that fit the chosen local-memory budget.\n2.  Form one score block and update stable running softmax statistics for the relevant query rows.\n3.  Accumulate weighted values while rescaling partial results when the running normalization changes.\n4.  Discard the temporary score block rather than writing the full score matrix to HBM.\n5.  Use the retained statistics and permitted recomputation for backward execution, then verify numerical outputs and measured traffic.\n\n## Science and assumptions\n\n### Score storage\n\n$$\nM_{\\mathrm{score}}=T_{\\mathrm{query}}T_{\\mathrm{key}}s\n$$\n\nA materialized dense score matrix contains query positions times key positions times bytes per score.\n\nSymbols and units\n\nM\\_score\n\nDense score-matrix payload for one head and request\n\nbytes\n\nT\\_query\n\nQuery-position count\n\npositions\n\nT\\_key\n\nKey-position count\n\npositions\n\ns\n\nStored bytes per score\n\nbytes/score\n\nTquery and Tkey count positions and s is bytes/score; Mscore is bytes for one dense score matrix.\n\nAssumptions One head and one request, no padding; causal storage and implementation fusion change actual allocation.\n\n## Engineering example\n\nSynthetic one-head score matrix with 4,096 queries, 4,096 keys and two-byte scores has 33,554,432 bytes, or 32 MiB. Avoiding its materialization removes one large temporary, while keys, values and outputs still require storage.\n\n## Distinctions and tradeoffs\n\nIO-aware exact attention reduces materialized intermediates; it does not by itself reduce the quadratic number of dense attention pair interactions.\n\nLess outer-memory traffic can justify additional recomputation and local bookkeeping; oversized tiles can exhaust scratchpad or reduce execution residency.\n\nLong contexts amplify intermediate-memory costs, so attention performance depends on IO organization as well as matrix arithmetic.\n\n## Specifications and interfaces\n\nScore shape\n\nTquery × Tkey\n\nElement count grows quadratically when both lengths scale together.\n\nExactness boundary\n\nSame mathematical attention with finite-precision effects\n\nOrder and rounding can differ from another implementation.\n\n### Interfaces\n\nQuery, key and value tiles Input\n\nBlocks staged from accelerator memory\n\nAttention output Output\n\nNormalized weighted values without a full stored score matrix\n\n## Check your understanding\n\n### What is the main avoided object in tiled IO-aware attention?\n\n1.  The full attention score/probability matrix in outer memory\n2.  All model weights\n3.  The causal relationship between tokens\n\nAnswer and reasoning\n\nThe full attention score/probability matrix in outer memory\n\nTiles and running normalization allow the result to be accumulated without retaining every pairwise score in HBM.\n\n## Evidence and scope\n\nThe example is a storage calculation, not a promised speedup. Exactness refers to the algorithm’s attention definition; finite-precision evaluation still needs testing.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/attention-io-tiling-430677d2c8cf857a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Kernel tiling, fusion and operand reuse](https://semiaifoundry.com/learn/atlas/kernel-locality-fusion/)\n\n### Related lessons\n\n-   [Occupancy and asynchronous tile pipelines](https://semiaifoundry.com/learn/atlas/gpu-occupancy-pipeline/)"
}