The mechanism
Self-attention computes content-dependent mixtures of information. Learned projections produce queries, keys, and values. Query–key dot products become scores; scaling, a causal mask when appropriate, and softmax turn them into weights. Those weights mix value vectors. Multiple heads can form different mixtures. A causal decoder prevents a position from using future tokens during training. The atlas displays a small matrix so the relationship between rows, columns, and weighted values remains visible.
Attention learning differentiates both the value path and the probabilities controlling that mixture. A causal mask removes future positions from the allowed softmax support; it is not merely a visual annotation. The executable decoder exposes projected Q/K/V, scores, probabilities and the complete gradient graph. A small analytical example clarifies why changing one score can redistribute weight across all allowed values.
Packing several examples into one input array adds a boundary condition to causal attention. In the corpus experience, a query may use only earlier or current positions from its own sampled document instance within the same block. Adjacent storage does not grant another document access to that context.
- Project representations into Q, K, and V.
- Form softmax(QKᵀ / √dₖ + mask).
- Multiply the weights by V and combine heads.
- Form scores only for positions the query may attend to.
- Normalize those scores together and mix their value vectors.
- Backpropagate through both softmax probabilities and value projections; verify prefix invariance under future-token changes.
- Use document-instance IDs as well as causal position order when constructing allowed attention keys.
- Keep shifted supervision separate from attention: a next-token label may be just outside the input block without becoming visible input context.
Science and assumptions
Scaled dot-product attention
Attention weights are a row-wise softmax of query–key scores divided by the square root of head width, plus a mask. Multiplying weights by values gives the output. Under the stated independent unit-variance coordinate assumptions, the unscaled dot-product variance is approximately head width.
Symbols and units
- Q
- Query matrix for one attention head
real T_q × d_k array
- K
- Key matrix
real T_k × d_k array
- V
- Value matrix
real T_k × d_v array
- A
- Row-normalized attention weights
dimensionless T_q × T_k array
- Y
- Mixed value output
real T_q × d_v array
- M
- Additive attention mask: zero or negative infinity
dimensionless score array
- d_k
- Query/key width
coordinates
- d_v
- Value width
coordinates
- T_q,T_k
- Query and key position counts
counts
- q,k
- Individual query/key vectors in the variance argument
dimensionless d_k-coordinate vectors
- Var
- Variance under the stated coordinate distribution
squared score units
Each row of A describes how one query mixes allowed value vectors. Dividing by √dₖ controls score scale: a sum of dₖ independent unit-variance products has variance dₖ. Without that adjustment, larger head widths can push softmax toward saturation. The matrix products expose why data reuse matters on GPU tensor units.
Assumptions The variance argument assumes independent, zero-mean, unit-variance query/key coordinates; trained vectors need not satisfy it exactly. M is zero for allowed positions and negative infinity for masked positions. This is one attention head before output projection.
Trainable attention from Q/K/V through gradients · worked relation
Trace a numerical softmax derivative through a causal value mixture.
Symbols and units
- o
- Scalar attention output in this example
value units
- a_j
- Softmax probability over allowed keys
dimensionless
- v_j
- Allowed scalar value
value units
- s_j
- Pre-softmax score
dimensionless
With allowed scores [0,ln3] and scalar values [2,6], attention probabilities are [0.25,0.75] and output is 5. For the output itself as the scalar objective, derivative with respect to score one is 0.25×(2−5)=−0.75 and score two is 0.75×(6−5)=0.75. At the first causal position only value 2 is allowed, so output is 2 regardless of future score or value.
Assumptions Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections.
Engineering example
For sequence length T, the unstructured score matrix has T² entries per head; memory-aware kernels change movement without changing the mathematical dependency.
With allowed scores [0,ln3] and scalar values [2,6], attention probabilities are [0.25,0.75] and output is 5. For the output itself as the scalar objective, derivative with respect to score one is 0.25×(2−5)=−0.75 and score two is 0.75×(6−5)=0.75. At the first causal position only value 2 is allowed, so output is 2 regardless of future score or value.
If the second document begins at block slot5, its query at slot7 can attend to slots5,6,7. Slots0–4 belong to the prior example and slots8 onward are future context. When an EOS is followed by another document’s BOS, the corresponding cross-document prediction loss is masked.
Distinctions and tradeoffs
Attention weights are not a complete explanation of a model’s reasoning.
Dense attention has quadratic pairwise work in sequence length. IO-aware kernels reduce memory traffic while implementing the same attention mathematics, subject to finite-precision differences.
Attention makes relationships within context available to each token.
Specifications and interfaces
- Dense score shape
- T × T per head
T is sequence length; algorithms need not store the full matrix.
- Causality
- Future positions masked
For an autoregressive decoder.
Interfaces
- Representations Input
- Queries, keys, values
- Residual stream Output
- Context-dependent value mixtures
Check your understanding
What does the attention weight matrix mix?
- CPU clocks
- Value vectors
- Raw electrical power
Answer and reasoning
Value vectors
Each row weights the value vectors available to that query.
What is the derivative of output with respect to the first score?
- −0.75.
- +0.75.
- Zero merely because attention probabilities sum to one.
Answer and reasoning
−0.75.
−0.75. Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections.
A packed query at slot7 belongs to a document starting at slot5. Which keys are allowed by this specimen?
- Every earlier slot0–7
- Only slots5–7
- Only the next token at slot8
Answer and reasoning
Only slots5–7
Both the same-instance constraint and causal order apply. Storage adjacency cannot override either boundary.
Historical context
2017 · The Transformer replaces recurrent sequence processing
Attention Is All You Need demonstrated an encoder–decoder built from attention and position-wise transformations, enabling parallel processing across training positions. It did not establish that attention alone constitutes intelligence.
Evidence and scope
The displayed equation is a model abstraction; masks and precision are explicit assumptions.
Mode: executable causal attention gradients. Scalar value example, softmax over the allowed set, values held fixed for the shown score derivative. Full model also differentiates values and projections. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
The experience constructs explicit masks and arrays. It does not execute fused variable-length attention or measure packing speedups. Documents crossing blocks lose earlier-block context in this declared specimen.
Inspect the causal mask and normalized score weights for hand-chosen vectors; attention temperature is separate from output sampling.