The mechanism
Positional schemes provide explicit order or distance information beyond token content and the attention mask. Position information can be added through embeddings or introduced into the attention computation. Rotary position embeddings rotate pairs of query and key coordinates as a function of position, allowing relative displacement to affect their dot product. This is a mathematical operation on vectors, not a rotating physical device. Position methods and long-context training choices vary across model families.
- Assign each token its position in the sequence.
- Apply the model’s chosen position transformation.
- Let attention scores depend on content and positional relationships.
Science and assumptions
Relative position from paired rotations
The dot product of a query rotated at m and key rotated at n equals the original query dotted with the key rotated by the relative offset n minus m.
Symbols and units
- R(p)
- Block-diagonal orthogonal rotary transform at position p
dimensionless d × d matrix
- q
- Query column vector before rotary transformation
real d × 1 vector
- k
- Key column vector before rotary transformation
real d × 1 vector
- m,n
- Query and key position indices
integer positions
- d
- Even rotary-vector dimension
coordinates
For rotary position embedding, R applies independent two-coordinate rotations. The query at position m and key at n are each transformed, but their dot product depends on the relative offset n−m. Position changes compatibility scores while retaining content in q and k.
Assumptions This identity uses matching orthogonal rotations and fixed frequencies on both vectors. It describes standard RoPE, not every positional method; context-extension schemes may modify position or frequency mappings.
Engineering example
Synthetic rotary pair starts with q = k = [1, 0]. Leave q at angle 0 and rotate k by π/2, giving [0, 1]; the dot product changes from 1 to 0. Rotating both pairs by the same angle preserves their relative-angle dot product.
Distinctions and tradeoffs
Rotary embeddings rotate numerical coordinates, not memory chips.
Extending context can require changes in training and position handling. Accepting a longer sequence is different from using all its information reliably.
Order distinguishes “dog bites person” from “person bites dog.”
Specifications and interfaces
- RoPE
- Pairwise coordinate rotation
One published approach; not universal.
- Length
- Training and implementation dependent
A long accepted context does not guarantee perfect recall.
Interfaces
- Sequence Input
- Token vectors and positions
- Attention Output
- Position-aware queries and keys
Check your understanding
Why add position information?
- To cool the chip
- To make token order available to the computation
- To turn every vector into a word
Answer and reasoning
To make token order available to the computation
Content alone does not specify where an item occurs in a sequence.
Historical context
2021 · RoFormer proposes rotary position embedding
The RoFormer preprint introduced RoPE, encoding position through query/key rotations so their interaction includes relative displacement.
Evidence and scope
The example illustrates the orthogonal-rotation identity in the linked RoFormer paper. Angles are teaching choices; frequency schedules, dimension pairing and context extension depend on the specified positional method.
Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.