{"id":"positional-encoding","level":"model","name":"Position representation","shortName":"Position representation","role":"Make sequence order available to attention.","description":"Positional schemes provide explicit order or distance information beyond token content and the attention mask. Position information can be added through embeddings or introduced into the attention computation. Rotary position embeddings rotate pairs of query and key coordinates as a function of position, allowing relative displacement to affect their dot product. This is a mathematical operation on vectors, not a rotating physical device. Position methods and long-context training choices vary across model families.","mechanism":["Assign each token its position in the sequence.","Apply the model’s chosen position transformation.","Let attention scores depend on content and positional relationships."],"specs":[{"label":"RoPE","value":"Pairwise coordinate rotation","context":"One published approach; not universal."},{"label":"Length","value":"Training and implementation dependent","context":"A long accepted context does not guarantee perfect recall."}],"interfaces":[{"name":"Sequence","direction":"in","signal":"Token vectors and positions"},{"name":"Attention","direction":"out","signal":"Position-aware queries and keys"}],"tradeoff":"Extending context can require changes in training and position handling. Accepting a longer sequence is different from using all its information reliably.","whyAI":"Order distinguishes “dog bites person” from “person bites dog.”","related":["embedding","attention","kv-cache"],"sources":[{"title":"Su et al. — RoFormer","url":"https://arxiv.org/abs/2104.09864"}],"scale":"Logical structure · mapped onto physical hardware","kind":"positional-encoding","misconception":"Rotary embeddings rotate numerical coordinates, not memory chips.","check":{"question":"Why add position information?","options":["To cool the chip","To make token order available to the computation","To turn every vector into a word"],"answer":1,"explanation":"Content alone does not specify where an item occurs in a sequence."},"branch":"Representation","subbranch":"Position and geometry","science":[{"title":"Relative position from paired rotations","equation":"(R(m)q)ᵀ(R(n)k) = qᵀR(n−m)k","explanation":"For rotary position embedding, R applies independent two-coordinate rotations. The query at position m and key at n are each transformed, but their dot product depends on the relative offset n−m. Position changes compatibility scores while retaining content in q and k.","assumptions":"This identity uses matching orthogonal rotations and fixed frequencies on both vectors. It describes standard RoPE, not every positional method; context-extension schemes may modify position or frequency mappings."}],"history":[{"year":2021,"title":"RoFormer proposes rotary position embedding","significance":"The RoFormer preprint introduced RoPE, encoding position through query/key rotations so their interaction includes relative displacement.","source":{"title":"Su et al. — RoFormer","url":"https://arxiv.org/abs/2104.09864"}}],"prerequisites":["embedding"],"learningObjective":"Use a two-coordinate rotation to distinguish vector position encoding from physical movement.","engineeringExample":"Synthetic rotary pair starts with q = k = [1, 0]. Leave q at angle 0 and rotate k by π/2, giving [0, 1]; the dot product changes from 1 to 0. Rotating both pairs by the same angle preserves their relative-angle dot product.","evidenceNotes":"The example illustrates the orthogonal-rotation identity in the linked RoFormer paper. Angles are teaching choices; frequency schedules, dimension pairing and context extension depend on the specified positional method.","labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}