Embedding table

The mechanism

An embedding table is a matrix with one row per vocabulary entry. Looking up an ID returns a vector that the rest of the network can transform. These values are learned during training. Related uses can acquire useful geometric relationships, but no single coordinate is guaranteed to have a neat human interpretation. The table lives in memory; its displayed three-dimensional form is a small sample of a much higher-dimensional array.

A lookup row is a shared parameter whenever the same token occurs more than once. Reverse-mode gradients from every occurrence must accumulate into that row. If input embeddings and the output head are tied, the output path adds another contribution to the same parameters. The executable lifecycle model uses separate matrices, so tying is a worked extension rather than an implicit property of all decoders.

Take a token ID as a row address.
Read the corresponding vector of learned values.
Pass the sequence of vectors into the model with position information.
Identify all input positions that read the same token row.
Add each position’s backward contribution at matching feature coordinates.
If weights are explicitly tied, add output-path gradients before the optimizer update.

Science and assumptions

Lookup is a sparse matrix multiplication

\begin{aligned}e_i&=\operatorname{onehot}(i)^{\mathsf T}E,\quad E\in\mathbb R^{V\times d}\\\frac{\partial\mathcal L}{\partial E_{j,:}}&=\sum_{t:\,\mathrm{token}_t=j}\frac{\partial\mathcal L}{\partial e_t}\end{aligned}

A one-hot row selects its embedding-table row. The lookup-path gradient for a shared vocabulary row sums gradients from every sequence position using that token.

The first e index names a vocabulary item; the gradient-sum e index names a sequence occurrence. The table explicitly distinguishes them. All values are model coordinates, not physical distances.

Symbols and units

E: Embedding table with one row per vocabulary item
real V × d array
V: Vocabulary size
count
d: Embedding width
coordinates
eᵢ: Row vector retrieved for token ID i in the lookup identity
real 1 × d vector
onehot(i): Column selector containing one at token ID i
real V × 1 vector
E_j,:: Embedding row for vocabulary ID j
real 1 × d vector
ℒ: Scalar training loss
dimensionless model loss
eₜ: Embedding occurrence at sequence position t in the gradient sum
real 1 × d vector
tokenₜ: Token ID at sequence position t
integer ID
i,j,t: Vocabulary or sequence indices as locally indicated
integer indices

A one-hot row selects exactly one of V embedding vectors, each with d coordinates. Implementations gather that row instead of materializing a giant one-hot matrix. During backpropagation, occurrences of the same token contribute to the gradient of the same shared row.

Assumptions The gradient expression covers the input lookup path only. A tied output projection, regularization, or other use of E can add gradients to additional rows; quantization changes physical storage.

Learned lookup gradients and tied weights · worked relation

\frac{\partial L}{\partial E_{v,:}}=\sum_{t:x_t=v}\frac{\partial L}{\partial e_t}+g^{tied\ output}_{v,:}

Accumulate gradients into shared lookup rows and distinguish tied from separate output weights.

Symbols and units

E_v: Shared embedding row for token v
dimensionless
e_t: Lookup result at input position t
dimensionless
g: Derivative contribution, zero here unless weights are explicitly tied
loss per parameter
x_t: Token identity
identifier
L: Scalar objective
loss

Token a appears twice and receives row gradients [1,2] and [3,−1]. Its input-embedding gradient is [4,1]. A separate token b receives [0.5,0.5] and must not be added to a. If an explicitly tied output path contributes [−2,1] to a, the combined tied gradient becomes [2,2]. The core instead keeps the output contribution in its separate unembedding tensor.

Assumptions Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate.

Engineering example

Synthetic table with 1,000 vocabulary rows, width 8 and two bytes per element uses 1,000 × 8 × 2 = 16,000 bytes. Looking up ID 17 returns one eight-element row, a 16-byte logical vector; it does not mean the token has semantic magnitude 17.

Distinctions and tradeoffs

The visible vector is not a literal concept stored in one neuron.

Wider vectors increase representation capacity and arithmetic/storage cost. Quantization can reduce bytes per value with numerical tradeoffs.

Embeddings connect symbolic input to the matrix computations executed by accelerators.

Specifications and interfaces

Shape: V × d
Vocabulary size V; vector width d.
Storage: V × d × bytes/value
Excludes quantization metadata and optimizer state.

Interfaces

Tokenizer Input: Integer IDs
Transformer blocks Output: Token vectors

Check your understanding

What does an embedding lookup return?

A row of learned numerical values
A network packet
A transistor address

Answer and reasoning

A row of learned numerical values

The ID indexes a trainable matrix row.

What is the input-only gradient for the repeated a row?

[3,−1], because the second occurrence replaces the first.
[4.5,1.5], because every token shares one row.
[4,1].

Answer and reasoning

[4,1].

[4,1]. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate.

Historical context

2003 · Distributed word representations in a neural language model

Bengio and colleagues jointly learned word vectors and a sequence probability model, showing how shared continuous representations could generalize beyond previously observed word sequences.

Bengio et al. — A Neural Probabilistic Language Model

Evidence and scope

This is dense logical storage and lookup as described by the linked embedding references. Padding, quantization metadata, tied output weights and training optimizer state are separate; no particular model vocabulary or width is assumed.

Mode: executable lookup gradients; worked tying. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.