{"id":"feed-forward","level":"model","name":"Feed-forward network","shortName":"Feed-forward network","role":"Transform features within each token position.","description":"A transformer also applies learned nonlinear transformations to each token representation. A feed-forward block usually expands or projects the vector, applies a nonlinearity or learned gate, and projects it back. It reuses the same parameters at each position. Gated variants such as SwiGLU use one branch to modulate another. The numerical arrays are large, so these transformations account for substantial matrix arithmetic and weight movement.\n\nA feed-forward sublayer transforms features independently at each token position using shared weights. Nonlinearity is what prevents two linear maps from collapsing into one equivalent linear map. Activation choice changes derivative behavior and parameter convention; a tanh block is not a ReLU or gated SwiGLU block. The lifecycle core uses tanh and explicit biases, all included in its parameter ledger.","mechanism":["Project a token vector into intermediate features.","Apply the nonlinearity or combine gated branches.","Project back to the residual width.","Compute the expanded affine features including bias.","Apply the declared nonlinearity coordinate by coordinate.","Project back to model width and accumulate shared weight gradients across tokens."],"specs":[{"label":"Input/output","value":"Usually width d","context":"Residual connections require compatible shapes."},{"label":"Intermediate width","value":"Architecture dependent","context":"No hidden Astra dimensions are assumed."}],"interfaces":[{"name":"Residual stream","direction":"in","signal":"Token feature vector"},{"name":"Residual stream","direction":"out","signal":"Transformed feature vector"}],"tradeoff":"Wider intermediates increase parameter storage and compute. Sparse expert routing changes which feed-forward weights are used per token.","whyAI":"Attention exchanges information; the feed-forward block transforms features.","related":["attention","residual-normalization","moe","tensor-core","kernel","reverse-mode-autodiff","tiny-causal-decoder"],"sources":[{"title":"Shazeer — GLU Variants Improve Transformer","url":"https://arxiv.org/abs/2002.05202"},{"title":"Attention Is All You Need","url":"https://arxiv.org/html/1706.03762v7"},{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"}],"scale":"Logical structure · mapped onto physical hardware","kind":"feed-forward","misconception":"A network layer is a logical operation; it is not a dedicated physical layer of silicon.","check":{"question":"Are feed-forward weights normally unique to each token position?","options":["Yes","No, the block reuses its weights across positions","Only on CPUs"],"answer":1,"explanation":"The same learned transformation is applied at different sequence positions."},"branch":"Transformer computation","subbranch":"Nonlinear feature transformation","science":[{"title":"SwiGLU and matrix cost","equation":"FFN(x) = [SiLU(xWg) ⊙ (xWu)]Wd; SiLU(z) = z·sigmoid(z); parameters ≈ 3d·dff","explanation":"Two projections create dff intermediate features. One branch gates the other coordinate by coordinate, and a third matrix projects back to width d. The three matrices explain both the parameter count and the large repeated matrix multiplications in a gated feed-forward block.","assumptions":"This is a bias-free SwiGLU example using row vectors, not a universal architecture specification. Wg and Wu have shape d×dff; Wd has shape dff×d. Nonlinear operations, optimizer state, and routing are excluded from the parameter estimate."},{"title":"Nonlinear features and block-level learning · worked relation","equation":"a=\\tanh(wx+b),\\qquad \\frac{\\partial a}{\\partial w}=(1-a^2)x,\\qquad P=dm+m+md+d","explanation":"For scalar h=tanh(wx+b), x=2, w=0 and b=0 give h=0 and dh/dw=(1−h²)x=2. At a saturated activation h≈0.99 the same derivative is about 0.0398. In the core, a 6→12→6 tanh block with both biases has 6×12+12+12×6+6=162 learned parameters. Its behavior must be evaluated through actual losses, not inferred from that count.","assumptions":"The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant."}],"history":[{"year":2020,"title":"GLU variants improve Transformer feed-forward blocks","significance":"Shazeer evaluated gated alternatives including SwiGLU and GEGLU in Transformer feed-forward sublayers, finding improvements over the tested ReLU and GELU baselines.","source":{"title":"Shazeer — GLU Variants Improve Transformer","url":"https://arxiv.org/abs/2002.05202"}}],"prerequisites":["embedding","mac"],"learningObjective":"Count the weight matrices and intermediate dimensions of a bias-free SwiGLU block.","engineeringExample":"Synthetic block with model width d = 4 and intermediate width dff = 12 has two 4 × 12 input projections and one 12 × 4 output projection. It stores 144 weights, or 288 bytes at two bytes per weight, before optimizer state, activations or metadata.\n\nFor scalar h=tanh(wx+b), x=2, w=0 and b=0 give h=0 and dh/dw=(1−h²)x=2. At a saturated activation h≈0.99 the same derivative is about 0.0398. In the core, a 6→12→6 tanh block with both biases has 6×12+12+12×6+6=162 learned parameters. Its behavior must be evaluated through actual losses, not inferred from that count.","evidenceNotes":"This uses the bias-free three-matrix SwiGLU form in the cited GLU paper. It is not a universal feed-forward architecture; activation work and runtime memory are not included in the parameter-byte count.\n\nMode: executable nonlinear block. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","checks":[{"question":"Why does the derivative shrink near h=0.99?","options":["Two linear maps are always equivalent to a nonlinear one.","The tanh derivative 1−h² is close to zero.","The optimizer automatically freezes every feature."],"answer":1,"explanation":"The tanh derivative 1−h² is close to zero. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant."}],"labId":"engineering","scope":"Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.","deepLabId":"footprint","investigationId":"execution","labScope":"Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled."}