{
  "schemaVersion": 1,
  "id": "atlas-lesson-positional-encoding",
  "title": "Position representation",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Make sequence order available to attention.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/positional-encoding/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Representation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The example illustrates the orthogonal-rotation identity in the linked RoFormer paper. Angles are teaching choices; frequency schedules, dimension pairing and context extension depend on the specified positional method.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/positional-encoding-d1cb9687bb660bb2.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#positional-encoding",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "70f6fefb27989a57614757e27402a6fff8cfeff41a6bbab988b545237429be4f",
  "contentLength": 4631,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-positional-encoding.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-positional-encoding.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Position representation",
      "start": 31,
      "end": 342
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 342,
      "end": 1063
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1063,
      "end": 1091
    },
    {
      "id": "section-4",
      "title": "Relative position from paired rotations",
      "start": 1091,
      "end": 2194
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2194,
      "end": 2447
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2447,
      "end": 2766
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2766,
      "end": 2983
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 2983,
      "end": 3095
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3095,
      "end": 3124
    },
    {
      "id": "section-10",
      "title": "Why add position information?",
      "start": 3124,
      "end": 3411
    },
    {
      "id": "section-11",
      "title": "Historical context",
      "start": 3411,
      "end": 3434
    },
    {
      "id": "section-12",
      "title": "2021 · RoFormer proposes rotary position embedding",
      "start": 3434,
      "end": 3687
    },
    {
      "id": "section-13",
      "title": "Evidence and scope",
      "start": 3687,
      "end": 4084
    },
    {
      "id": "section-14",
      "title": "Sources",
      "start": 4084,
      "end": 4356
    },
    {
      "id": "section-15",
      "title": "Continue exploring",
      "start": 4356,
      "end": 4379
    },
    {
      "id": "section-16",
      "title": "Foundations",
      "start": 4379,
      "end": 4469
    },
    {
      "id": "section-17",
      "title": "Related lessons",
      "start": 4469,
      "end": 4631
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#positional-encoding"
    },
    {
      "title": "Su et al. — RoFormer",
      "url": "https://arxiv.org/abs/2104.09864"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/positional-encoding-d1cb9687bb660bb2.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    }
  ],
  "recordHash": "af435ab5ea03980a075a6819e560e07e228918aa37b40dd3be0f09072468fe42",
  "contentFormat": "text/markdown",
  "content": "Representation · Atlas v1.3.0\n\n# Position representation\n\nMake sequence order available to attention.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#positional-encoding)Logical structure · mapped onto physical hardware\n\nLearning objectiveUse a two-coordinate rotation to distinguish vector position encoding from physical movement.\n\n## The mechanism\n\nPositional schemes provide explicit order or distance information beyond token content and the attention mask. Position information can be added through embeddings or introduced into the attention computation. Rotary position embeddings rotate pairs of query and key coordinates as a function of position, allowing relative displacement to affect their dot product. This is a mathematical operation on vectors, not a rotating physical device. Position methods and long-context training choices vary across model families.\n\n1.  Assign each token its position in the sequence.\n2.  Apply the model’s chosen position transformation.\n3.  Let attention scores depend on content and positional relationships.\n\n## Science and assumptions\n\n### Relative position from paired rotations\n\n$$\n(R(m)q)^{\\mathsf T}(R(n)k)=q^{\\mathsf T}R(n-m)k\n$$\n\nThe dot product of a query rotated at m and key rotated at n equals the original query dotted with the key rotated by the relative offset n minus m.\n\nSymbols and units\n\nR(p)\n\nBlock-diagonal orthogonal rotary transform at position p\n\ndimensionless d × d matrix\n\nq\n\nQuery column vector before rotary transformation\n\nreal d × 1 vector\n\nk\n\nKey column vector before rotary transformation\n\nreal d × 1 vector\n\nm,n\n\nQuery and key position indices\n\ninteger positions\n\nd\n\nEven rotary-vector dimension\n\ncoordinates\n\nFor rotary position embedding, R applies independent two-coordinate rotations. The query at position m and key at n are each transformed, but their dot product depends on the relative offset n−m. Position changes compatibility scores while retaining content in q and k.\n\nAssumptions This identity uses matching orthogonal rotations and fixed frequencies on both vectors. It describes standard RoPE, not every positional method; context-extension schemes may modify position or frequency mappings.\n\n## Engineering example\n\nSynthetic rotary pair starts with q = k = \\[1, 0\\]. Leave q at angle 0 and rotate k by π/2, giving \\[0, 1\\]; the dot product changes from 1 to 0. Rotating both pairs by the same angle preserves their relative-angle dot product.\n\n## Distinctions and tradeoffs\n\nRotary embeddings rotate numerical coordinates, not memory chips.\n\nExtending context can require changes in training and position handling. Accepting a longer sequence is different from using all its information reliably.\n\nOrder distinguishes “dog bites person” from “person bites dog.”\n\n## Specifications and interfaces\n\nRoPE\n\nPairwise coordinate rotation\n\nOne published approach; not universal.\n\nLength\n\nTraining and implementation dependent\n\nA long accepted context does not guarantee perfect recall.\n\n### Interfaces\n\nSequence Input\n\nToken vectors and positions\n\nAttention Output\n\nPosition-aware queries and keys\n\n## Check your understanding\n\n### Why add position information?\n\n1.  To cool the chip\n2.  To make token order available to the computation\n3.  To turn every vector into a word\n\nAnswer and reasoning\n\nTo make token order available to the computation\n\nContent alone does not specify where an item occurs in a sequence.\n\n## Historical context\n\n### 2021 · RoFormer proposes rotary position embedding\n\nThe RoFormer preprint introduced RoPE, encoding position through query/key rotations so their interaction includes relative displacement.\n\n[Su et al. — RoFormer](https://arxiv.org/abs/2104.09864)\n\n## Evidence and scope\n\nThe example illustrates the orthogonal-rotation identity in the linked RoFormer paper. Angles are teaching choices; frequency schedules, dimension pairing and context extension depend on the specified positional method.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [Su et al. — RoFormer](https://arxiv.org/abs/2104.09864)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/positional-encoding-d1cb9687bb660bb2.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n\n### Related lessons\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)"
}