{"id":"embedding","level":"model","name":"Embedding table","shortName":"Embedding table","role":"Give discrete tokens a trainable numerical representation.","description":"An embedding table is a matrix with one row per vocabulary entry. Looking up an ID returns a vector that the rest of the network can transform. These values are learned during training. Related uses can acquire useful geometric relationships, but no single coordinate is guaranteed to have a neat human interpretation. The table lives in memory; its displayed three-dimensional form is a small sample of a much higher-dimensional array.\n\nA lookup row is a shared parameter whenever the same token occurs more than once. Reverse-mode gradients from every occurrence must accumulate into that row. If input embeddings and the output head are tied, the output path adds another contribution to the same parameters. The executable lifecycle model uses separate matrices, so tying is a worked extension rather than an implicit property of all decoders.","mechanism":["Take a token ID as a row address.","Read the corresponding vector of learned values.","Pass the sequence of vectors into the model with position information.","Identify all input positions that read the same token row.","Add each position’s backward contribution at matching feature coordinates.","If weights are explicitly tied, add output-path gradients before the optimizer update."],"specs":[{"label":"Shape","value":"V × d","context":"Vocabulary size V; vector width d."},{"label":"Storage","value":"V × d × bytes/value","context":"Excludes quantization metadata and optimizer state."}],"interfaces":[{"name":"Tokenizer","direction":"in","signal":"Integer IDs"},{"name":"Transformer blocks","direction":"out","signal":"Token vectors"}],"tradeoff":"Wider vectors increase representation capacity and arithmetic/storage cost. Quantization can reduce bytes per value with numerical tradeoffs.","whyAI":"Embeddings connect symbolic input to the matrix computations executed by accelerators.","related":["tokenizer","positional-encoding","attention","hbm-stack","reverse-mode-autodiff","tiny-causal-decoder"],"sources":[{"title":"PyTorch — Embedding","url":"https://docs.pytorch.org/docs/2.14/generated/torch.nn.Embedding.html"},{"title":"Bengio et al. — A Neural Probabilistic Language Model","url":"https://www.jmlr.org/papers/v3/bengio03a.html"},{"title":"Attention Is All You Need","url":"https://arxiv.org/html/1706.03762v7"}],"scale":"Logical structure · mapped onto physical hardware","kind":"embedding","misconception":"The visible vector is not a literal concept stored in one neuron.","check":{"question":"What does an embedding lookup return?","options":["A row of learned numerical values","A network packet","A transistor address"],"answer":0,"explanation":"The ID indexes a trainable matrix row."},"branch":"Representation","subbranch":"Learned vectors","science":[{"title":"Lookup is a sparse matrix multiplication","equation":"e_i = one_hot(i)ᵀ E; E ∈ ℝ^(V×d); ∂ℒ/∂E_j = Σ[t: token_t=j] ∂ℒ/∂e_t","explanation":"A one-hot row selects exactly one of V embedding vectors, each with d coordinates. Implementations gather that row instead of materializing a giant one-hot matrix. During backpropagation, occurrences of the same token contribute to the gradient of the same shared row.","assumptions":"The gradient expression covers the input lookup path only. A tied output projection, regularization, or other use of E can add gradients to additional rows; quantization changes physical storage."},{"title":"Learned lookup gradients and tied weights · worked relation","equation":"\\frac{\\partial L}{\\partial E_{v,:}}=\\sum_{t:x_t=v}\\frac{\\partial L}{\\partial e_t}+g^{tied\\ output}_{v,:}","explanation":"Token a appears twice and receives row gradients [1,2] and [3,−1]. Its input-embedding gradient is [4,1]. A separate token b receives [0.5,0.5] and must not be added to a. If an explicitly tied output path contributes [−2,1] to a, the combined tied gradient becomes [2,2]. The core instead keeps the output contribution in its separate unembedding tensor.","assumptions":"Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate."}],"history":[{"year":2003,"title":"Distributed word representations in a neural language model","significance":"Bengio and colleagues jointly learned word vectors and a sequence probability model, showing how shared continuous representations could generalize beyond previously observed word sequences.","source":{"title":"Bengio et al. — A Neural Probabilistic Language Model","url":"https://www.jmlr.org/papers/v3/bengio03a.html"}}],"prerequisites":["tokenizer"],"learningObjective":"Compute table storage and explain how an ID selects a vector without creating meaning from its numeric value.","engineeringExample":"Synthetic table with 1,000 vocabulary rows, width 8 and two bytes per element uses 1,000 × 8 × 2 = 16,000 bytes. Looking up ID 17 returns one eight-element row, a 16-byte logical vector; it does not mean the token has semantic magnitude 17.\n\nToken a appears twice and receives row gradients [1,2] and [3,−1]. Its input-embedding gradient is [4,1]. A separate token b receives [0.5,0.5] and must not be added to a. If an explicitly tied output path contributes [−2,1] to a, the combined tied gradient becomes [2,2]. The core instead keeps the output contribution in its separate unembedding tensor.","evidenceNotes":"This is dense logical storage and lookup as described by the linked embedding references. Padding, quantization metadata, tied output weights and training optimizer state are separate; no particular model vocabulary or width is assumed.\n\nMode: executable lookup gradients; worked tying. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","checks":[{"question":"What is the input-only gradient for the repeated a row?","options":["[3,−1], because the second occurrence replaces the first.","[4.5,1.5], because every token shares one row.","[4,1]."],"answer":2,"explanation":"[4,1]. Matching coordinates, summed path derivatives and an explicit tied/separate convention. Core weights are separate."}],"labId":"engineering","scope":"Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.","deepLabId":"footprint","investigationId":"execution","labScope":"Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled."}