Feed-forward network

The mechanism

A transformer also applies learned nonlinear transformations to each token representation. A feed-forward block usually expands or projects the vector, applies a nonlinearity or learned gate, and projects it back. It reuses the same parameters at each position. Gated variants such as SwiGLU use one branch to modulate another. The numerical arrays are large, so these transformations account for substantial matrix arithmetic and weight movement.

A feed-forward sublayer transforms features independently at each token position using shared weights. Nonlinearity is what prevents two linear maps from collapsing into one equivalent linear map. Activation choice changes derivative behavior and parameter convention; a tanh block is not a ReLU or gated SwiGLU block. The lifecycle core uses tanh and explicit biases, all included in its parameter ledger.

Project a token vector into intermediate features.
Apply the nonlinearity or combine gated branches.
Project back to the residual width.
Compute the expanded affine features including bias.
Apply the declared nonlinearity coordinate by coordinate.
Project back to model width and accumulate shared weight gradients across tokens.

Science and assumptions

SwiGLU and matrix cost

\begin{aligned}\operatorname{FFN}(x)&=[\operatorname{SiLU}(xW_g)\odot(xW_u)]W_d\\\operatorname{SiLU}(z)&=z\,\operatorname{sigmoid}(z),\qquad N_{\mathrm{parameters}}\approx3d\,d_{\mathrm{ff}}\end{aligned}

SwiGLU multiplies the SiLU gate projection elementwise by the up projection, then applies the down projection. SiLU is z times sigmoid of z. The three bias-free matrices contain approximately three times model width times feed-forward width parameters.

Symbols and units

x: Input row vector
real 1 × d vector
W_g,W_u: Gate and up-projection weights
real d × d_ff matrices
W_d: Down-projection weights
real d_ff × d matrix
d: Model representation width
coordinates
d_ff: Intermediate feed-forward width
coordinates
z: Scalar or elementwise nonlinear input
dimensionless model coordinates
⊙: Elementwise multiplication
operator
N_parameters: Bias-free three-matrix parameter count
parameters
FFN(x): Output of the selected SwiGLU block
real 1 × d vector

Two projections create dff intermediate features. One branch gates the other coordinate by coordinate, and a third matrix projects back to width d. The three matrices explain both the parameter count and the large repeated matrix multiplications in a gated feed-forward block.

Assumptions This is a bias-free SwiGLU example using row vectors, not a universal architecture specification. Wg and Wu have shape d×dff; Wd has shape dff×d. Nonlinear operations, optimizer state, and routing are excluded from the parameter estimate.

Nonlinear features and block-level learning · worked relation

a=\tanh(wx+b),\qquad \frac{\partial a}{\partial w}=(1-a^2)x,\qquad P=dm+m+md+d

Count a nonlinear block’s learned state and differentiate its activation.

Symbols and units

a: Scalar tanh activation
dimensionless
w,b: Scalar weight and bias
dimensionless
x: Scalar input
dimensionless
d,m: Model and feed-forward widths in the separate parameter ledger
features
P: Two affine maps plus biases
parameters

For scalar h=tanh(wx+b), x=2, w=0 and b=0 give h=0 and dh/dw=(1−h²)x=2. At a saturated activation h≈0.99 the same derivative is about 0.0398. In the core, a 6→12→6 tanh block with both biases has 6×12+12+12×6+6=162 learned parameters. Its behavior must be evaluated through actual losses, not inferred from that count.

Assumptions The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant.

Engineering example

Synthetic block with model width d = 4 and intermediate width dff = 12 has two 4 × 12 input projections and one 12 × 4 output projection. It stores 144 weights, or 288 bytes at two bytes per weight, before optimizer state, activations or metadata.

Distinctions and tradeoffs

A network layer is a logical operation; it is not a dedicated physical layer of silicon.

Wider intermediates increase parameter storage and compute. Sparse expert routing changes which feed-forward weights are used per token.

Attention exchanges information; the feed-forward block transforms features.

Specifications and interfaces

Input/output: Usually width d
Residual connections require compatible shapes.
Intermediate width: Architecture dependent
No hidden Astra dimensions are assumed.

Interfaces

Residual stream Input: Token feature vector
Residual stream Output: Transformed feature vector

Check your understanding

Are feed-forward weights normally unique to each token position?

Yes
No, the block reuses its weights across positions
Only on CPUs

Answer and reasoning

No, the block reuses its weights across positions

The same learned transformation is applied at different sequence positions.

Why does the derivative shrink near h=0.99?

Two linear maps are always equivalent to a nonlinear one.
The tanh derivative 1−h² is close to zero.
The optimizer automatically freezes every feature.

Answer and reasoning

The tanh derivative 1−h² is close to zero.

The tanh derivative 1−h² is close to zero. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant.

Historical context

2020 · GLU variants improve Transformer feed-forward blocks

Shazeer evaluated gated alternatives including SwiGLU and GEGLU in Transformer feed-forward sublayers, finding improvements over the tested ReLU and GELU baselines.

Shazeer — GLU Variants Improve Transformer

Evidence and scope

This uses the bias-free three-matrix SwiGLU form in the cited GLU paper. It is not a universal feed-forward architecture; activation work and runtime memory are not included in the parameter-byte count.

Mode: executable nonlinear block. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Budget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.