# Position representation Canonical: https://semiaifoundry.com/learn/atlas/positional-encoding/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Representation · Atlas v1.3.0 # Position representation Make sequence order available to attention. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#positional-encoding)Logical structure · mapped onto physical hardware Learning objectiveUse a two-coordinate rotation to distinguish vector position encoding from physical movement. ## The mechanism Positional schemes provide explicit order or distance information beyond token content and the attention mask. Position information can be added through embeddings or introduced into the attention computation. Rotary position embeddings rotate pairs of query and key coordinates as a function of position, allowing relative displacement to affect their dot product. This is a mathematical operation on vectors, not a rotating physical device. Position methods and long-context training choices vary across model families. 1. Assign each token its position in the sequence. 2. Apply the model’s chosen position transformation. 3. Let attention scores depend on content and positional relationships. ## Science and assumptions ### Relative position from paired rotations $$ (R(m)q)^{\mathsf T}(R(n)k)=q^{\mathsf T}R(n-m)k $$ The dot product of a query rotated at m and key rotated at n equals the original query dotted with the key rotated by the relative offset n minus m. Symbols and units R(p) Block-diagonal orthogonal rotary transform at position p dimensionless d × d matrix q Query column vector before rotary transformation real d × 1 vector k Key column vector before rotary transformation real d × 1 vector m,n Query and key position indices integer positions d Even rotary-vector dimension coordinates For rotary position embedding, R applies independent two-coordinate rotations. The query at position m and key at n are each transformed, but their dot product depends on the relative offset n−m. Position changes compatibility scores while retaining content in q and k. Assumptions This identity uses matching orthogonal rotations and fixed frequencies on both vectors. It describes standard RoPE, not every positional method; context-extension schemes may modify position or frequency mappings. ## Engineering example Synthetic rotary pair starts with q = k = \[1, 0\]. Leave q at angle 0 and rotate k by π/2, giving \[0, 1\]; the dot product changes from 1 to 0. Rotating both pairs by the same angle preserves their relative-angle dot product. ## Distinctions and tradeoffs Rotary embeddings rotate numerical coordinates, not memory chips. Extending context can require changes in training and position handling. Accepting a longer sequence is different from using all its information reliably. Order distinguishes “dog bites person” from “person bites dog.” ## Specifications and interfaces RoPE Pairwise coordinate rotation One published approach; not universal. Length Training and implementation dependent A long accepted context does not guarantee perfect recall. ### Interfaces Sequence Input Token vectors and positions Attention Output Position-aware queries and keys ## Check your understanding ### Why add position information? 1. To cool the chip 2. To make token order available to the computation 3. To turn every vector into a word Answer and reasoning To make token order available to the computation Content alone does not specify where an item occurs in a sequence. ## Historical context ### 2021 · RoFormer proposes rotary position embedding The RoFormer preprint introduced RoPE, encoding position through query/key rotations so their interaction includes relative displacement. [Su et al. — RoFormer](https://arxiv.org/abs/2104.09864) ## Evidence and scope The example illustrates the orthogonal-rotation identity in the linked RoFormer paper. Angles are teaching choices; frequency schedules, dimension pairing and context extension depend on the specified positional method. Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism. ## Sources - [Su et al. — RoFormer](https://arxiv.org/abs/2104.09864) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/positional-encoding-d1cb9687bb660bb2.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/) ### Related lessons - [Self-attention](https://semiaifoundry.com/learn/atlas/attention/) - [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)