# Embedding table Canonical: https://semiaifoundry.com/learn/atlas/embedding/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Representation · Atlas v1.3.0 # Embedding table Give discrete tokens a trainable numerical representation. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#embedding)Logical structure · mapped onto physical hardware Learning objectiveCompute table storage and explain how an ID selects a vector without creating meaning from its numeric value. ## The mechanism An embedding table is a matrix with one row per vocabulary entry. Looking up an ID returns a vector that the rest of the network can transform. These values are learned during training. Related uses can acquire useful geometric relationships, but no single coordinate is guaranteed to have a neat human interpretation. The table lives in memory; its displayed three-dimensional form is a small sample of a much higher-dimensional array. A lookup row is a shared parameter whenever the same token occurs more than once. Reverse-mode gradients from every occurrence must accumulate into that row. If input embeddings and the output head are tied, the output path adds another contribution to the same parameters. The executable lifecycle model uses separate matrices, so tying is a worked extension rather than an implicit property of all decoders. 1. Take a token ID as a row address. 2. Read the corresponding vector of learned values. 3. Pass the sequence of vectors into the model with position information. 4. Identify all input positions that read the same token row. 5. Add each position’s backward contribution at matching feature coordinates. 6. If weights are explicitly tied, add output-path gradients before the optimizer update. ## Science and assumptions ### Lookup is a sparse matrix multiplication $$ \begin{aligned}e_i&=\operatorname{onehot}(i)^{\mathsf T}E,\quad E\in\mathbb R^{V\times d}\\\frac{\partial\mathcal L}{\partial E_{j,:}}&=\sum_{t:\,\mathrm{token}_t=j}\frac{\partial\mathcal L}{\partial e_t}\end{aligned} $$ A one-hot row selects its embedding-table row. The lookup-path gradient for a shared vocabulary row sums gradients from every sequence position using that token. The first e index names a vocabulary item; the gradient-sum e index names a sequence occurrence. The table explicitly distinguishes them. All values are model coordinates, not physical distances. Symbols and units E Embedding table with one row per vocabulary item real V × d array V Vocabulary size count d Embedding width coordinates eᵢ Row vector retrieved for token ID i in the lookup identity real 1 × d vector onehot(i) Column selector containing one at token ID i real V × 1 vector E\_j,: Embedding row for vocabulary ID j real 1 × d vector ℒ Scalar training loss dimensionless model loss eₜ Embedding occurrence at sequence position t in the gradient sum real 1 × d vector tokenₜ Token ID at sequence position t integer ID i,j,t Vocabulary or sequence indices as locally indicated integer indices A one-hot row selects exactly one of V embedding vectors, each with d coordinates. Implementations gather that row instead of materializing a giant one-hot matrix. During backpropagation, occurrences of the same token contribute to the gradient of the same shared row. Assumptions The gradient expression covers the input lookup path only. A tied output projection, regularization, or other use of E can add gradients to additional rows; quantization changes physical storage. ### Learned lookup gradients and tied weights · worked relation $$ \frac{\partial L}{\partial E_{v,:}}=\sum_{t:x_t=v}\frac{\partial L}{\partial e_t}+g^{tied\ output}_{v,:} $$ Accumulate gradients into shared lookup rows and distinguish tied from separate output weights. Symbols and units E\_v Shared embedding row for token v dimensionless e\_t Lookup result at input position t dimensionless g Derivative contribution, zero here unless weights are explicitly tied loss per parameter x\_t Token identity identifier L Scalar objective loss Token a appears twice and receives row gradients \[1,2\] and \[3,−1\]. Its input-embedding gradient is \[4,1\]. A separate token b receives \[0.5,0.5\] and must not be added to a. If an explicitly tied output path contributes \[−2,1\] to a, the combined tied gradient becomes \[2,2\]. The core instead keeps the output contribution in its separate unembedding tensor. Assumptions Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. ## Engineering example Synthetic table with 1,000 vocabulary rows, width 8 and two bytes per element uses 1,000 × 8 × 2 = 16,000 bytes. Looking up ID 17 returns one eight-element row, a 16-byte logical vector; it does not mean the token has semantic magnitude 17. Token a appears twice and receives row gradients \[1,2\] and \[3,−1\]. Its input-embedding gradient is \[4,1\]. A separate token b receives \[0.5,0.5\] and must not be added to a. If an explicitly tied output path contributes \[−2,1\] to a, the combined tied gradient becomes \[2,2\]. The core instead keeps the output contribution in its separate unembedding tensor. ## Distinctions and tradeoffs The visible vector is not a literal concept stored in one neuron. Wider vectors increase representation capacity and arithmetic/storage cost. Quantization can reduce bytes per value with numerical tradeoffs. Embeddings connect symbolic input to the matrix computations executed by accelerators. ## Specifications and interfaces Shape V × d Vocabulary size V; vector width d. Storage V × d × bytes/value Excludes quantization metadata and optimizer state. ### Interfaces Tokenizer Input Integer IDs Transformer blocks Output Token vectors ## Check your understanding ### What does an embedding lookup return? 1. A row of learned numerical values 2. A network packet 3. A transistor address Answer and reasoning A row of learned numerical values The ID indexes a trainable matrix row. ### What is the input-only gradient for the repeated a row? 1. \[3,−1\], because the second occurrence replaces the first. 2. \[4.5,1.5\], because every token shares one row. 3. \[4,1\]. Answer and reasoning \[4,1\]. \[4,1\]. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. ## Historical context ### 2003 · Distributed word representations in a neural language model Bengio and colleagues jointly learned word vectors and a sequence probability model, showing how shared continuous representations could generalize beyond previously observed word sequences. [Bengio et al. — A Neural Probabilistic Language Model](https://www.jmlr.org/papers/v3/bengio03a.html) ## Evidence and scope This is dense logical storage and lookup as described by the linked embedding references. Padding, quantization metadata, tied output weights and training optimizer state are separate; no particular model vocabulary or width is assumed. Mode: executable lookup gradients; worked tying. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled. ## Sources - [PyTorch — Embedding](https://docs.pytorch.org/docs/2.14/generated/torch.nn.Embedding.html) - [Bengio et al. — A Neural Probabilistic Language Model](https://www.jmlr.org/papers/v3/bengio03a.html) - [Attention Is All You Need](https://arxiv.org/html/1706.03762v7) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/embedding-a409279536f75f65.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/) ### Related lessons - [Position representation](https://semiaifoundry.com/learn/atlas/positional-encoding/) - [Self-attention](https://semiaifoundry.com/learn/atlas/attention/) - [High-bandwidth memory stack](https://semiaifoundry.com/learn/atlas/hbm-stack/) - [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/) - [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)