{
  "schemaVersion": 1,
  "id": "atlas-lesson-embedding",
  "title": "Embedding table",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Give discrete tokens a trainable numerical representation.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/embedding/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Representation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This is dense logical storage and lookup as described by the linked embedding references. Padding, quantization metadata, tied output weights and training optimizer state are separate; no particular model vocabulary or width is assumed.\n\nMode: executable lookup gradients; worked tying. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/embedding-a409279536f75f65.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#embedding",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "05fee4f87241174adb89653dd91bf341496b28af0f3e450ae88ee66069851646",
  "contentLength": 8509,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-embedding.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-embedding.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Embedding table",
      "start": 31,
      "end": 355
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 355,
      "end": 1622
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1622,
      "end": 1650
    },
    {
      "id": "section-4",
      "title": "Lookup is a sparse matrix multiplication",
      "start": 1650,
      "end": 3400
    },
    {
      "id": "section-5",
      "title": "Learned lookup gradients and tied weights · worked relation",
      "start": 3400,
      "end": 4456
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4456,
      "end": 5091
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5091,
      "end": 5420
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5420,
      "end": 5587
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5587,
      "end": 5675
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5675,
      "end": 5704
    },
    {
      "id": "section-11",
      "title": "What does an embedding lookup return?",
      "start": 5704,
      "end": 5929
    },
    {
      "id": "section-12",
      "title": "What is the input-only gradient for the repeated a row?",
      "start": 5929,
      "end": 6278
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 6278,
      "end": 6301
    },
    {
      "id": "section-14",
      "title": "2003 · Distributed word representations in a neural language model",
      "start": 6301,
      "end": 6669
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 6669,
      "end": 7449
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 7449,
      "end": 7922
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 7922,
      "end": 7945
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 7945,
      "end": 8029
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 8029,
      "end": 8509
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#embedding"
    },
    {
      "title": "Bengio et al. — A Neural Probabilistic Language Model",
      "url": "https://www.jmlr.org/papers/v3/bengio03a.html"
    },
    {
      "title": "PyTorch — Embedding",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.nn.Embedding.html"
    },
    {
      "title": "Attention Is All You Need",
      "url": "https://arxiv.org/html/1706.03762v7"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/embedding-a409279536f75f65.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Tokenizer",
      "url": "https://semiaifoundry.com/learn/atlas/tokenizer/"
    },
    {
      "title": "Position representation",
      "url": "https://semiaifoundry.com/learn/atlas/positional-encoding/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "High-bandwidth memory stack",
      "url": "https://semiaifoundry.com/learn/atlas/hbm-stack/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "370be89c7395df098bf7ce9b9c63bc7a2f45d9f43ec3fd5aec9fc85db16da3f0",
  "contentFormat": "text/markdown",
  "content": "Representation · Atlas v1.3.0\n\n# Embedding table\n\nGive discrete tokens a trainable numerical representation.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#embedding)Logical structure · mapped onto physical hardware\n\nLearning objectiveCompute table storage and explain how an ID selects a vector without creating meaning from its numeric value.\n\n## The mechanism\n\nAn embedding table is a matrix with one row per vocabulary entry. Looking up an ID returns a vector that the rest of the network can transform. These values are learned during training. Related uses can acquire useful geometric relationships, but no single coordinate is guaranteed to have a neat human interpretation. The table lives in memory; its displayed three-dimensional form is a small sample of a much higher-dimensional array.\n\nA lookup row is a shared parameter whenever the same token occurs more than once. Reverse-mode gradients from every occurrence must accumulate into that row. If input embeddings and the output head are tied, the output path adds another contribution to the same parameters. The executable lifecycle model uses separate matrices, so tying is a worked extension rather than an implicit property of all decoders.\n\n1.  Take a token ID as a row address.\n2.  Read the corresponding vector of learned values.\n3.  Pass the sequence of vectors into the model with position information.\n4.  Identify all input positions that read the same token row.\n5.  Add each position’s backward contribution at matching feature coordinates.\n6.  If weights are explicitly tied, add output-path gradients before the optimizer update.\n\n## Science and assumptions\n\n### Lookup is a sparse matrix multiplication\n\n$$\n\\begin{aligned}e_i&=\\operatorname{onehot}(i)^{\\mathsf T}E,\\quad E\\in\\mathbb R^{V\\times d}\\\\\\frac{\\partial\\mathcal L}{\\partial E_{j,:}}&=\\sum_{t:\\,\\mathrm{token}_t=j}\\frac{\\partial\\mathcal L}{\\partial e_t}\\end{aligned}\n$$\n\nA one-hot row selects its embedding-table row. The lookup-path gradient for a shared vocabulary row sums gradients from every sequence position using that token.\n\nThe first e index names a vocabulary item; the gradient-sum e index names a sequence occurrence. The table explicitly distinguishes them. All values are model coordinates, not physical distances.\n\nSymbols and units\n\nE\n\nEmbedding table with one row per vocabulary item\n\nreal V × d array\n\nV\n\nVocabulary size\n\ncount\n\nd\n\nEmbedding width\n\ncoordinates\n\neᵢ\n\nRow vector retrieved for token ID i in the lookup identity\n\nreal 1 × d vector\n\nonehot(i)\n\nColumn selector containing one at token ID i\n\nreal V × 1 vector\n\nE\\_j,:\n\nEmbedding row for vocabulary ID j\n\nreal 1 × d vector\n\nℒ\n\nScalar training loss\n\ndimensionless model loss\n\neₜ\n\nEmbedding occurrence at sequence position t in the gradient sum\n\nreal 1 × d vector\n\ntokenₜ\n\nToken ID at sequence position t\n\ninteger ID\n\ni,j,t\n\nVocabulary or sequence indices as locally indicated\n\ninteger indices\n\nA one-hot row selects exactly one of V embedding vectors, each with d coordinates. Implementations gather that row instead of materializing a giant one-hot matrix. During backpropagation, occurrences of the same token contribute to the gradient of the same shared row.\n\nAssumptions The gradient expression covers the input lookup path only. A tied output projection, regularization, or other use of E can add gradients to additional rows; quantization changes physical storage.\n\n### Learned lookup gradients and tied weights · worked relation\n\n$$\n\\frac{\\partial L}{\\partial E_{v,:}}=\\sum_{t:x_t=v}\\frac{\\partial L}{\\partial e_t}+g^{tied\\ output}_{v,:}\n$$\n\nAccumulate gradients into shared lookup rows and distinguish tied from separate output weights.\n\nSymbols and units\n\nE\\_v\n\nShared embedding row for token v\n\ndimensionless\n\ne\\_t\n\nLookup result at input position t\n\ndimensionless\n\ng\n\nDerivative contribution, zero here unless weights are explicitly tied\n\nloss per parameter\n\nx\\_t\n\nToken identity\n\nidentifier\n\nL\n\nScalar objective\n\nloss\n\nToken a appears twice and receives row gradients \\[1,2\\] and \\[3,−1\\]. Its input-embedding gradient is \\[4,1\\]. A separate token b receives \\[0.5,0.5\\] and must not be added to a. If an explicitly tied output path contributes \\[−2,1\\] to a, the combined tied gradient becomes \\[2,2\\]. The core instead keeps the output contribution in its separate unembedding tensor.\n\nAssumptions Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate.\n\n## Engineering example\n\nSynthetic table with 1,000 vocabulary rows, width 8 and two bytes per element uses 1,000 × 8 × 2 = 16,000 bytes. Looking up ID 17 returns one eight-element row, a 16-byte logical vector; it does not mean the token has semantic magnitude 17.\n\nToken a appears twice and receives row gradients \\[1,2\\] and \\[3,−1\\]. Its input-embedding gradient is \\[4,1\\]. A separate token b receives \\[0.5,0.5\\] and must not be added to a. If an explicitly tied output path contributes \\[−2,1\\] to a, the combined tied gradient becomes \\[2,2\\]. The core instead keeps the output contribution in its separate unembedding tensor.\n\n## Distinctions and tradeoffs\n\nThe visible vector is not a literal concept stored in one neuron.\n\nWider vectors increase representation capacity and arithmetic/storage cost. Quantization can reduce bytes per value with numerical tradeoffs.\n\nEmbeddings connect symbolic input to the matrix computations executed by accelerators.\n\n## Specifications and interfaces\n\nShape\n\nV × d\n\nVocabulary size V; vector width d.\n\nStorage\n\nV × d × bytes/value\n\nExcludes quantization metadata and optimizer state.\n\n### Interfaces\n\nTokenizer Input\n\nInteger IDs\n\nTransformer blocks Output\n\nToken vectors\n\n## Check your understanding\n\n### What does an embedding lookup return?\n\n1.  A row of learned numerical values\n2.  A network packet\n3.  A transistor address\n\nAnswer and reasoning\n\nA row of learned numerical values\n\nThe ID indexes a trainable matrix row.\n\n### What is the input-only gradient for the repeated a row?\n\n1.  \\[3,−1\\], because the second occurrence replaces the first.\n2.  \\[4.5,1.5\\], because every token shares one row.\n3.  \\[4,1\\].\n\nAnswer and reasoning\n\n\\[4,1\\].\n\n\\[4,1\\]. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate.\n\n## Historical context\n\n### 2003 · Distributed word representations in a neural language model\n\nBengio and colleagues jointly learned word vectors and a sequence probability model, showing how shared continuous representations could generalize beyond previously observed word sequences.\n\n[Bengio et al. — A Neural Probabilistic Language Model](https://www.jmlr.org/papers/v3/bengio03a.html)\n\n## Evidence and scope\n\nThis is dense logical storage and lookup as described by the linked embedding references. Padding, quantization metadata, tied output weights and training optimizer state are separate; no particular model vocabulary or width is assumed.\n\nMode: executable lookup gradients; worked tying. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nBudget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.\n\n## Sources\n\n-   [PyTorch — Embedding](https://docs.pytorch.org/docs/2.14/generated/torch.nn.Embedding.html)\n-   [Bengio et al. — A Neural Probabilistic Language Model](https://www.jmlr.org/papers/v3/bengio03a.html)\n-   [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/embedding-a409279536f75f65.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n\n### Related lessons\n\n-   [Position representation](https://semiaifoundry.com/learn/atlas/positional-encoding/)\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}