{ "schemaVersion": 1, "id": "atlas-lesson-attention", "title": "Self-attention", "category": "learning", "format": "Atlas lesson", "description": "Exchange information between token representations.", "canonicalUrl": "https://semiaifoundry.com/learn/atlas/attention/", "publishedDate": null, "revisionDate": "2026-10-07", "author": null, "topic": "Transformer computation", "series": null, "license": null, "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html", "version": "1.3.0", "status": "Published educational lesson", "evidenceScope": "The displayed equation is a model abstraction; masks and precision are explicit assumptions.\n\nMode: executable causal attention gradients. Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThe experience constructs explicit masks and arrays. It does not execute fused variable-length attention or measure packing speedups. Documents crossing blocks lose earlier-block context in this declared specimen.", "contentCoverage": "published-lesson", "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/attention-ca04a2b8e0cce22b.json", "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#attention", "related": [], "supersedes": [], "supersededBy": [], "contentHash": "fc3fc3accb8714f89fdfda9af895c77b27760534a788bb34f0a17c9df13e9821", "contentLength": 10251, "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-attention.json", "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-attention.md", "sections": [ { "id": "section-1", "title": "Self-attention", "start": 40, "end": 307 }, { "id": "section-2", "title": "The mechanism", "start": 307, "end": 2206 }, { "id": "section-3", "title": "Science and assumptions", "start": 2206, "end": 2234 }, { "id": "section-4", "title": "Scaled dot-product attention", "start": 2234, "end": 4005 }, { "id": "section-5", "title": "Trainable attention from Q/K/V through gradients · worked relation", "start": 4005, "end": 4968 }, { "id": "section-6", "title": "Engineering example", "start": 4968, "end": 5793 }, { "id": "section-7", "title": "Distinctions and tradeoffs", "start": 5793, "end": 6165 }, { "id": "section-8", "title": "Specifications and interfaces", "start": 6165, "end": 6368 }, { "id": "section-9", "title": "Interfaces", "start": 6368, "end": 6488 }, { "id": "section-10", "title": "Check your understanding", "start": 6488, "end": 6517 }, { "id": "section-11", "title": "What does the attention weight matrix mix?", "start": 6517, "end": 6722 }, { "id": "section-12", "title": "What is the derivative of output with respect to the first score?", "start": 6722, "end": 7071 }, { "id": "section-13", "title": "A packed query at slot7 belongs to a document starting at slot5. Which keys are allowed by this specimen?", "start": 7071, "end": 7407 }, { "id": "section-14", "title": "Historical context", "start": 7407, "end": 7430 }, { "id": "section-15", "title": "2017 · The Transformer replaces recurrent sequence processing", "start": 7430, "end": 7816 }, { "id": "section-16", "title": "Evidence and scope", "start": 7816, "end": 8690 }, { "id": "section-17", "title": "Sources", "start": 8690, "end": 9251 }, { "id": "section-18", "title": "Continue exploring", "start": 9251, "end": 9274 }, { "id": "section-19", "title": "Foundations", "start": 9274, "end": 9545 }, { "id": "section-20", "title": "Related lessons", "start": 9545, "end": 10251 } ], "citations": [ { "title": "Explore in Atlas", "url": "https://semiaifoundry.com/ai-atlas/#attention" }, { "title": "Vaswani et al. — Attention Is All You Need", "url": "https://arxiv.org/abs/1706.03762" }, { "title": "Dao et al. — FlashAttention", "url": "https://arxiv.org/abs/2205.14135" }, { "title": "Attention Is All You Need", "url": "https://arxiv.org/html/1706.03762v7" }, { "title": "Megatron Bridge 0.3.1 · packed sequence boundaries", "url": "https://docs.nvidia.com/nemo/megatron-bridge/0.3.1/training/packed-sequences.html" }, { "title": "Published lesson data", "url": "https://semiaifoundry.com/ai-atlas/lesson-data/attention-ca04a2b8e0cce22b.json" }, { "title": "Atlas releases and licensing", "url": "https://semiaifoundry.com/ai-atlas/licenses.html" }, { "title": "Embedding table", "url": "https://semiaifoundry.com/learn/atlas/embedding/" }, { "title": "Position representation", "url": "https://semiaifoundry.com/learn/atlas/positional-encoding/" }, { "title": "Multiply–accumulate and fused arithmetic", "url": "https://semiaifoundry.com/learn/atlas/mac/" }, { "title": "Feed-forward network", "url": "https://semiaifoundry.com/learn/atlas/feed-forward/" }, { "title": "Tensor Core matrix unit", "url": "https://semiaifoundry.com/learn/atlas/tensor-core/" }, { "title": "Key–value cache", "url": "https://semiaifoundry.com/learn/atlas/kv-cache/" }, { "title": "A differentiable computation graph", "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/" }, { "title": "Assemble an initialized causal language model", "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/" }, { "title": "Training data pipeline", "url": "https://semiaifoundry.com/learn/atlas/training-data/" }, { "title": "Tokenizer", "url": "https://semiaifoundry.com/learn/atlas/tokenizer/" }, { "title": "Loss, gradients & optimizer", "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/" } ], "recordHash": "2a95e250ab7f7566e26201ff1b766d6c3304b3910fea02f8ae784e756b5ce63d", "contentFormat": "text/markdown", "content": "Transformer computation · Atlas v1.3.0\n\n# Self-attention\n\nExchange information between token representations.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#attention)Logical structure · mapped onto physical hardware\n\nLearning objectiveRelate attention scores to a quadratic sequence interaction.\n\n## The mechanism\n\nSelf-attention computes content-dependent mixtures of information. Learned projections produce queries, keys, and values. Query–key dot products become scores; scaling, a causal mask when appropriate, and softmax turn them into weights. Those weights mix value vectors. Multiple heads can form different mixtures. A causal decoder prevents a position from using future tokens during training. The atlas displays a small matrix so the relationship between rows, columns, and weighted values remains visible.\n\nAttention learning differentiates both the value path and the probabilities controlling that mixture. A causal mask removes future positions from the allowed softmax support; it is not merely a visual annotation. The executable decoder exposes projected Q/K/V, scores, probabilities and the complete gradient graph. A small analytical example clarifies why changing one score can redistribute weight across all allowed values.\n\nPacking several examples into one input array adds a boundary condition to causal attention. In the corpus experience, a query may use only earlier or current positions from its own sampled document instance within the same block. Adjacent storage does not grant another document access to that context.\n\n1. Project representations into Q, K, and V.\n2. Form softmax(QKᵀ / √dₖ + mask).\n3. Multiply the weights by V and combine heads.\n4. Form scores only for positions the query may attend to.\n5. Normalize those scores together and mix their value vectors.\n6. Backpropagate through both softmax probabilities and value projections; verify prefix invariance under future-token changes.\n7. Use document-instance IDs as well as causal position order when constructing allowed attention keys.\n8. Keep shifted supervision separate from attention: a next-token label may be just outside the input block without becoming visible input context.\n\n## Science and assumptions\n\n### Scaled dot-product attention\n\n$$\n\\begin{aligned}A&=\\operatorname{softmax}_{\\mathrm{rows}}\\!\\left(\\frac{QK^{\\mathsf T}}{\\sqrt{d_k}}+M\\right),\\qquad Y=AV\\\\\\operatorname{Var}(q^{\\mathsf T}k)&\\approx d_k\\end{aligned}\n$$\n\nAttention weights are a row-wise softmax of query–key scores divided by the square root of head width, plus a mask. Multiplying weights by values gives the output. Under the stated independent unit-variance coordinate assumptions, the unscaled dot-product variance is approximately head width.\n\nSymbols and units\n\nQ\n\nQuery matrix for one attention head\n\nreal T\\_q × d\\_k array\n\nK\n\nKey matrix\n\nreal T\\_k × d\\_k array\n\nV\n\nValue matrix\n\nreal T\\_k × d\\_v array\n\nA\n\nRow-normalized attention weights\n\ndimensionless T\\_q × T\\_k array\n\nY\n\nMixed value output\n\nreal T\\_q × d\\_v array\n\nM\n\nAdditive attention mask: zero or negative infinity\n\ndimensionless score array\n\nd\\_k\n\nQuery/key width\n\ncoordinates\n\nd\\_v\n\nValue width\n\ncoordinates\n\nT\\_q,T\\_k\n\nQuery and key position counts\n\ncounts\n\nq,k\n\nIndividual query/key vectors in the variance argument\n\ndimensionless d\\_k-coordinate vectors\n\nVar\n\nVariance under the stated coordinate distribution\n\nsquared score units\n\nEach row of A describes how one query mixes allowed value vectors. Dividing by √dₖ controls score scale: a sum of dₖ independent unit-variance products has variance dₖ. Without that adjustment, larger head widths can push softmax toward saturation. The matrix products expose why data reuse matters on GPU tensor units.\n\nAssumptions The variance argument assumes independent, zero-mean, unit-variance query/key coordinates; trained vectors need not satisfy it exactly. M is zero for allowed positions and negative infinity for masked positions. This is one attention head before output projection.\n\n### Trainable attention from Q/K/V through gradients · worked relation\n\n$$\no=\\sum_j a_jv_j,\\qquad \\frac{\\partial o}{\\partial s_j}=a_j(v_j-o)\n$$\n\nTrace a numerical softmax derivative through a causal value mixture.\n\nSymbols and units\n\no\n\nScalar attention output in this example\n\nvalue units\n\na\\_j\n\nSoftmax probability over allowed keys\n\ndimensionless\n\nv\\_j\n\nAllowed scalar value\n\nvalue units\n\ns\\_j\n\nPre-softmax score\n\ndimensionless\n\nWith allowed scores \\[0,ln3\\] and scalar values \\[2,6\\], attention probabilities are \\[0.25,0.75\\] and output is 5. For the output itself as the scalar objective, derivative with respect to score one is 0.25×(2−5)=−0.75 and score two is 0.75×(6−5)=0.75. At the first causal position only value 2 is allowed, so output is 2 regardless of future score or value.\n\nAssumptions Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections.\n\n## Engineering example\n\nFor sequence length T, the unstructured score matrix has T² entries per head; memory-aware kernels change movement without changing the mathematical dependency.\n\nWith allowed scores \\[0,ln3\\] and scalar values \\[2,6\\], attention probabilities are \\[0.25,0.75\\] and output is 5. For the output itself as the scalar objective, derivative with respect to score one is 0.25×(2−5)=−0.75 and score two is 0.75×(6−5)=0.75. At the first causal position only value 2 is allowed, so output is 2 regardless of future score or value.\n\nIf the second document begins at block slot5, its query at slot7 can attend to slots5,6,7. Slots0–4 belong to the prior example and slots8 onward are future context. When an EOS is followed by another document’s BOS, the corresponding cross-document prediction loss is masked.\n\n## Distinctions and tradeoffs\n\nAttention weights are not a complete explanation of a model’s reasoning.\n\nDense attention has quadratic pairwise work in sequence length. IO-aware kernels reduce memory traffic while implementing the same attention mathematics, subject to finite-precision differences.\n\nAttention makes relationships within context available to each token.\n\n## Specifications and interfaces\n\nDense score shape\n\nT × T per head\n\nT is sequence length; algorithms need not store the full matrix.\n\nCausality\n\nFuture positions masked\n\nFor an autoregressive decoder.\n\n### Interfaces\n\nRepresentations Input\n\nQueries, keys, values\n\nResidual stream Output\n\nContext-dependent value mixtures\n\n## Check your understanding\n\n### What does the attention weight matrix mix?\n\n1. CPU clocks\n2. Value vectors\n3. Raw electrical power\n\nAnswer and reasoning\n\nValue vectors\n\nEach row weights the value vectors available to that query.\n\n### What is the derivative of output with respect to the first score?\n\n1. −0.75.\n2. +0.75.\n3. Zero merely because attention probabilities sum to one.\n\nAnswer and reasoning\n\n−0.75.\n\n−0.75. Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections.\n\n### A packed query at slot7 belongs to a document starting at slot5. Which keys are allowed by this specimen?\n\n1. Every earlier slot0–7\n2. Only slots5–7\n3. Only the next token at slot8\n\nAnswer and reasoning\n\nOnly slots5–7\n\nBoth the same-instance constraint and causal order apply. Storage adjacency cannot override either boundary.\n\n## Historical context\n\n### 2017 · The Transformer replaces recurrent sequence processing\n\nAttention Is All You Need demonstrated an encoder–decoder built from attention and position-wise transformations, enabling parallel processing across training positions. It did not establish that attention alone constitutes intelligence.\n\n[Vaswani et al. — Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n\n## Evidence and scope\n\nThe displayed equation is a model abstraction; masks and precision are explicit assumptions.\n\nMode: executable causal attention gradients. Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nThe experience constructs explicit masks and arrays. It does not execute fused variable-length attention or measure packing speedups. Documents crossing blocks lose earlier-block context in this declared specimen.\n\nInspect the causal mask and normalized score weights for hand-chosen vectors; attention temperature is separate from output sampling.\n\n## Sources\n\n- [Vaswani et al. — Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n- [Dao et al. — FlashAttention](https://arxiv.org/abs/2205.14135)\n- [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n- [Megatron Bridge 0.3.1 · packed sequence boundaries](https://docs.nvidia.com/nemo/megatron-bridge/0.3.1/training/packed-sequences.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/attention-ca04a2b8e0cce22b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n- [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n- [Position representation](https://semiaifoundry.com/learn/atlas/positional-encoding/)\n- [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n\n### Related lessons\n\n- [Feed-forward network](https://semiaifoundry.com/learn/atlas/feed-forward/)\n- [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/)\n- [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n- [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n- [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)\n- [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n- [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n- [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)" }