{
  "schemaVersion": 1,
  "id": "atlas-lesson-feed-forward",
  "title": "Feed-forward network",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Transform features within each token position.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/feed-forward/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Transformer computation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This uses the bias-free three-matrix SwiGLU form in the cited GLU paper. It is not a universal feed-forward architecture; activation work and runtime memory are not included in the parameter-byte count.\n\nMode: executable nonlinear block. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/feed-forward-c10ceb758cac6b13.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#feed-forward",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "424359c791ea7426b57a08e80fbc916e71d03e7c07f727d18dce3c3e3bb5755b",
  "contentLength": 8554,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-feed-forward.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-feed-forward.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Feed-forward network",
      "start": 40,
      "end": 333
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 333,
      "end": 1563
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1563,
      "end": 1591
    },
    {
      "id": "section-4",
      "title": "SwiGLU and matrix cost",
      "start": 1591,
      "end": 3139
    },
    {
      "id": "section-5",
      "title": "Nonlinear features and block-level learning · worked relation",
      "start": 3139,
      "end": 4122
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4122,
      "end": 4715
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 4715,
      "end": 5052
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5052,
      "end": 5251
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5251,
      "end": 5364
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5364,
      "end": 5393
    },
    {
      "id": "section-11",
      "title": "Are feed-forward weights normally unique to each token position?",
      "start": 5393,
      "end": 5693
    },
    {
      "id": "section-12",
      "title": "Why does the derivative shrink near h=0.99?",
      "start": 5693,
      "end": 6179
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 6179,
      "end": 6202
    },
    {
      "id": "section-14",
      "title": "2020 · GLU variants improve Transformer feed-forward blocks",
      "start": 6202,
      "end": 6513
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 6513,
      "end": 7291
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 7291,
      "end": 7707
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 7707,
      "end": 7730
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 7730,
      "end": 7911
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 7911,
      "end": 8554
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#feed-forward"
    },
    {
      "title": "Shazeer — GLU Variants Improve Transformer",
      "url": "https://arxiv.org/abs/2002.05202"
    },
    {
      "title": "Attention Is All You Need",
      "url": "https://arxiv.org/html/1706.03762v7"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/feed-forward-c10ceb758cac6b13.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Multiply–accumulate and fused arithmetic",
      "url": "https://semiaifoundry.com/learn/atlas/mac/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Residual stream & normalization",
      "url": "https://semiaifoundry.com/learn/atlas/residual-normalization/"
    },
    {
      "title": "Mixture of experts",
      "url": "https://semiaifoundry.com/learn/atlas/moe/"
    },
    {
      "title": "Tensor Core matrix unit",
      "url": "https://semiaifoundry.com/learn/atlas/tensor-core/"
    },
    {
      "title": "GPU kernel and thread hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/kernel/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "9d709e34c9b93abc99cd23457deb17944c0d4dc59f063a30ac67dc8f78907d03",
  "contentFormat": "text/markdown",
  "content": "Transformer computation · Atlas v1.3.0\n\n# Feed-forward network\n\nTransform features within each token position.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#feed-forward)Logical structure · mapped onto physical hardware\n\nLearning objectiveCount the weight matrices and intermediate dimensions of a bias-free SwiGLU block.\n\n## The mechanism\n\nA transformer also applies learned nonlinear transformations to each token representation. A feed-forward block usually expands or projects the vector, applies a nonlinearity or learned gate, and projects it back. It reuses the same parameters at each position. Gated variants such as SwiGLU use one branch to modulate another. The numerical arrays are large, so these transformations account for substantial matrix arithmetic and weight movement.\n\nA feed-forward sublayer transforms features independently at each token position using shared weights. Nonlinearity is what prevents two linear maps from collapsing into one equivalent linear map. Activation choice changes derivative behavior and parameter convention; a tanh block is not a ReLU or gated SwiGLU block. The lifecycle core uses tanh and explicit biases, all included in its parameter ledger.\n\n1.  Project a token vector into intermediate features.\n2.  Apply the nonlinearity or combine gated branches.\n3.  Project back to the residual width.\n4.  Compute the expanded affine features including bias.\n5.  Apply the declared nonlinearity coordinate by coordinate.\n6.  Project back to model width and accumulate shared weight gradients across tokens.\n\n## Science and assumptions\n\n### SwiGLU and matrix cost\n\n$$\n\\begin{aligned}\\operatorname{FFN}(x)&=[\\operatorname{SiLU}(xW_g)\\odot(xW_u)]W_d\\\\\\operatorname{SiLU}(z)&=z\\,\\operatorname{sigmoid}(z),\\qquad N_{\\mathrm{parameters}}\\approx3d\\,d_{\\mathrm{ff}}\\end{aligned}\n$$\n\nSwiGLU multiplies the SiLU gate projection elementwise by the up projection, then applies the down projection. SiLU is z times sigmoid of z. The three bias-free matrices contain approximately three times model width times feed-forward width parameters.\n\nSymbols and units\n\nx\n\nInput row vector\n\nreal 1 × d vector\n\nW\\_g,W\\_u\n\nGate and up-projection weights\n\nreal d × d\\_ff matrices\n\nW\\_d\n\nDown-projection weights\n\nreal d\\_ff × d matrix\n\nd\n\nModel representation width\n\ncoordinates\n\nd\\_ff\n\nIntermediate feed-forward width\n\ncoordinates\n\nz\n\nScalar or elementwise nonlinear input\n\ndimensionless model coordinates\n\n⊙\n\nElementwise multiplication\n\noperator\n\nN\\_parameters\n\nBias-free three-matrix parameter count\n\nparameters\n\nFFN(x)\n\nOutput of the selected SwiGLU block\n\nreal 1 × d vector\n\nTwo projections create dff intermediate features. One branch gates the other coordinate by coordinate, and a third matrix projects back to width d. The three matrices explain both the parameter count and the large repeated matrix multiplications in a gated feed-forward block.\n\nAssumptions This is a bias-free SwiGLU example using row vectors, not a universal architecture specification. Wg and Wu have shape d×dff; Wd has shape dff×d. Nonlinear operations, optimizer state, and routing are excluded from the parameter estimate.\n\n### Nonlinear features and block-level learning · worked relation\n\n$$\na=\\tanh(wx+b),\\qquad \\frac{\\partial a}{\\partial w}=(1-a^2)x,\\qquad P=dm+m+md+d\n$$\n\nCount a nonlinear block’s learned state and differentiate its activation.\n\nSymbols and units\n\na\n\nScalar tanh activation\n\ndimensionless\n\nw,b\n\nScalar weight and bias\n\ndimensionless\n\nx\n\nScalar input\n\ndimensionless\n\nd,m\n\nModel and feed-forward widths in the separate parameter ledger\n\nfeatures\n\nP\n\nTwo affine maps plus biases\n\nparameters\n\nFor scalar h=tanh(wx+b), x=2, w=0 and b=0 give h=0 and dh/dw=(1−h²)x=2. At a saturated activation h≈0.99 the same derivative is about 0.0398. In the core, a 6→12→6 tanh block with both biases has 6×12+12+12×6+6=162 learned parameters. Its behavior must be evaluated through actual losses, not inferred from that count.\n\nAssumptions The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant.\n\n## Engineering example\n\nSynthetic block with model width d = 4 and intermediate width dff = 12 has two 4 × 12 input projections and one 12 × 4 output projection. It stores 144 weights, or 288 bytes at two bytes per weight, before optimizer state, activations or metadata.\n\nFor scalar h=tanh(wx+b), x=2, w=0 and b=0 give h=0 and dh/dw=(1−h²)x=2. At a saturated activation h≈0.99 the same derivative is about 0.0398. In the core, a 6→12→6 tanh block with both biases has 6×12+12+12×6+6=162 learned parameters. Its behavior must be evaluated through actual losses, not inferred from that count.\n\n## Distinctions and tradeoffs\n\nA network layer is a logical operation; it is not a dedicated physical layer of silicon.\n\nWider intermediates increase parameter storage and compute. Sparse expert routing changes which feed-forward weights are used per token.\n\nAttention exchanges information; the feed-forward block transforms features.\n\n## Specifications and interfaces\n\nInput/output\n\nUsually width d\n\nResidual connections require compatible shapes.\n\nIntermediate width\n\nArchitecture dependent\n\nNo hidden Astra dimensions are assumed.\n\n### Interfaces\n\nResidual stream Input\n\nToken feature vector\n\nResidual stream Output\n\nTransformed feature vector\n\n## Check your understanding\n\n### Are feed-forward weights normally unique to each token position?\n\n1.  Yes\n2.  No, the block reuses its weights across positions\n3.  Only on CPUs\n\nAnswer and reasoning\n\nNo, the block reuses its weights across positions\n\nThe same learned transformation is applied at different sequence positions.\n\n### Why does the derivative shrink near h=0.99?\n\n1.  Two linear maps are always equivalent to a nonlinear one.\n2.  The tanh derivative 1−h² is close to zero.\n3.  The optimizer automatically freezes every feature.\n\nAnswer and reasoning\n\nThe tanh derivative 1−h² is close to zero.\n\nThe tanh derivative 1−h² is close to zero. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant.\n\n## Historical context\n\n### 2020 · GLU variants improve Transformer feed-forward blocks\n\nShazeer evaluated gated alternatives including SwiGLU and GEGLU in Transformer feed-forward sublayers, finding improvements over the tested ReLU and GELU baselines.\n\n[Shazeer — GLU Variants Improve Transformer](https://arxiv.org/abs/2002.05202)\n\n## Evidence and scope\n\nThis uses the bias-free three-matrix SwiGLU form in the cited GLU paper. It is not a universal feed-forward architecture; activation work and runtime memory are not included in the parameter-byte count.\n\nMode: executable nonlinear block. The scalar activation symbol h and hidden-width ledger are explicitly separated by context. Core uses tanh, not the original Transformer ReLU or a gated variant. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nBudget weights, KV and optional optimizer/master state; activations, gradients and the numerical training or quantization algorithm are not modeled.\n\n## Sources\n\n-   [Shazeer — GLU Variants Improve Transformer](https://arxiv.org/abs/2002.05202)\n-   [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/feed-forward-c10ceb758cac6b13.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Multiply–accumulate and fused arithmetic](https://semiaifoundry.com/learn/atlas/mac/)\n\n### Related lessons\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)\n-   [Mixture of experts](https://semiaifoundry.com/learn/atlas/moe/)\n-   [Tensor Core matrix unit](https://semiaifoundry.com/learn/atlas/tensor-core/)\n-   [GPU kernel and thread hierarchy](https://semiaifoundry.com/learn/atlas/kernel/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}