{
  "schemaVersion": 1,
  "id": "atlas-lesson-residual-normalization",
  "title": "Residual stream & normalization",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Keep a deep sequence of transformations trainable.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/residual-normalization/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Transformer computation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The example uses the linked LayerNorm definition over two features. RMSNorm, pre/post-normalization placement and learned parameters differ; the residual operands must have matching shapes, and neither operation guarantees stable training.\n\nMode: executable residual and RMS paths. Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/residual-normalization-89e7d8a2d9d3801e.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#residual-normalization",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "183e209e2f1bb41bd00738258a1c736f698e3494b6062970946cf8c61b7df853",
  "contentLength": 9207,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-residual-normalization.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-residual-normalization.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Residual stream & normalization",
      "start": 40,
      "end": 365
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 365,
      "end": 1837
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1837,
      "end": 1865
    },
    {
      "id": "section-4",
      "title": "Normalization and a direct gradient route",
      "start": 1865,
      "end": 3869
    },
    {
      "id": "section-5",
      "title": "Activation scale and gradient flow · worked relation",
      "start": 3869,
      "end": 4836
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4836,
      "end": 5464
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5464,
      "end": 5773
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5773,
      "end": 5952
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5952,
      "end": 6055
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 6055,
      "end": 6084
    },
    {
      "id": "section-11",
      "title": "What travels through a residual connection?",
      "start": 6084,
      "end": 6351
    },
    {
      "id": "section-12",
      "title": "Does RMS normalization force the output mean to zero?",
      "start": 6351,
      "end": 6845
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 6845,
      "end": 6868
    },
    {
      "id": "section-14",
      "title": "2016 · Layer normalization removes dependence on batch statistics",
      "start": 6868,
      "end": 7218
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 7218,
      "end": 8022
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 8022,
      "end": 8619
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 8619,
      "end": 8642
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 8642,
      "end": 8812
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 8812,
      "end": 9207
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#residual-normalization"
    },
    {
      "title": "Ba et al. — Layer Normalization",
      "url": "https://arxiv.org/abs/1607.06450"
    },
    {
      "title": "PyTorch — LayerNorm",
      "url": "https://docs.pytorch.org/docs/2.14/generated/torch.nn.LayerNorm.html"
    },
    {
      "title": "He et al. — Deep Residual Learning",
      "url": "https://arxiv.org/abs/1512.03385"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Root Mean Square Layer Normalization",
      "url": "https://arxiv.org/html/1910.07467v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/residual-normalization-89e7d8a2d9d3801e.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Feed-forward network",
      "url": "https://semiaifoundry.com/learn/atlas/feed-forward/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "11eecfc7ef5797501edce7ba4904f49dd77061a92a819b0aa070740eed9e1163",
  "contentFormat": "text/markdown",
  "content": "Transformer computation · Atlas v1.3.0\n\n# Residual stream & normalization\n\nKeep a deep sequence of transformations trainable.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#residual-normalization)Logical structure · mapped onto physical hardware\n\nLearning objectiveCompute a small LayerNorm example and distinguish normalization from a residual addition.\n\n## The mechanism\n\nResidual connections add a block’s output to a continuing representation. This lets each block contribute an update and provides a direct route through a deep computation graph. Normalization controls aspects of numerical scale; LayerNorm uses the mean and variance over specified feature dimensions, while other normalizers differ. Placement before or after sublayers matters. Neither operation creates persistent autobiographical memory: the stream is an intermediate activation for the current computation.\n\nA residual addition passes an input directly to the output alongside a learned transformation, creating an identity term in the derivative. Normalization modifies feature scale before or after a sublayer according to the declared architecture. RMS normalization does not subtract the mean, and its learned gain and epsilon conventions must be stated. The lifecycle variant uses pre-RMS normalization with fixed gain one and epsilon 1e-5; it is not exactly the original Transformer normalization placement.\n\n1.  Normalize the features at the architecture’s chosen location.\n2.  Compute the block’s transformation.\n3.  Add the compatible result to the continuing representation.\n4.  Trace the identity branch separately from the sublayer branch.\n5.  Compute mean square over the declared feature axis, with epsilon placed explicitly.\n6.  Compare actual activation and gradient values rather than assuming a residual guarantees stable learning.\n\n## Science and assumptions\n\n### Normalization and a direct gradient route\n\n$$\n\\begin{aligned}\\operatorname{LN}(x)&=\\gamma\\odot\\frac{x-\\mu}{\\sqrt{\\sigma^2+\\varepsilon}}+\\beta\\\\y&=x+F(x),\\qquad\\frac{\\partial y}{\\partial x}=I+J_F(x)\\end{aligned}\n$$\n\nLayerNorm subtracts the feature mean and divides by the square root of variance plus stabilizer, then applies learned scale and shift. Separately, an identity residual add has Jacobian equal to the identity plus the transformation Jacobian.\n\nNormalization and residual addition are separate relations, not a claim that every architecture uses this combined block. Model coordinates are usually represented as dimensionless numbers.\n\nSymbols and units\n\nx\n\nInput representation over the normalized feature set\n\nreal d-coordinate vector\n\nμ\n\nMean of x over those features\n\nsame coordinate units as x\n\nσ²\n\nPopulation variance over those features\n\nsquared coordinate units\n\nε\n\nPositive normalization stabilizer\n\nsame units as σ²\n\nγ\n\nLearned elementwise output scale\n\noutput-coordinate units\n\nβ\n\nLearned elementwise output shift\n\noutput-coordinate units\n\ny\n\nResidual-block output\n\nsame shape and coordinate units as x\n\nF(x)\n\nResidual transformation with matching output shape\n\nsame shape and coordinate units as x\n\nI\n\nIdentity Jacobian for the skip path\n\ndimensionless d × d matrix\n\nJ\\_F(x)\n\nJacobian of F with respect to x\n\ndimensionless d × d matrix\n\n⊙\n\nElementwise multiplication\n\noperator\n\nLayerNorm measures the mean and variance of one representation across its normalized features, then applies learned scale and shift. Separately, a residual add supplies an identity term in the Jacobian, giving gradients a direct path alongside the learned transformation. Neither guarantees that all deep networks train stably.\n\nAssumptions The residual equation describes an identity skip with matching shapes; F may include normalization. LayerNorm uses population variance over the chosen features. RMSNorm omits mean subtraction, and pre-norm versus post-norm changes the full block Jacobian.\n\n### Activation scale and gradient flow · worked relation\n\n$$\n\\frac{dy}{dx}=1+f^{\\prime}(x),\\qquad \\operatorname{RMSNorm}(x)_i=\\frac{x_i}{\\sqrt{d^{-1}\\sum_j x_j^2+\\epsilon}}\n$$\n\nDistinguish an identity gradient path from normalization’s scale transformation.\n\nSymbols and units\n\nx\\_i\n\nInput feature\n\ndimensionless\n\nd\n\nFeatures normalized together\n\nfeatures\n\nepsilon\n\nPositive stabilizer in mean-square units\n\ndimensionless squared\n\nf\n\nResidual sublayer\n\nfunction\n\ny\n\nResidual output\n\ndimensionless\n\nFor y=x+f(x), local derivative is 1+f′(x); if f′(x)=−0.2 the total is 0.8. For vector \\[3,4\\], RMS without epsilon is sqrt((9+16)/2)=sqrt12.5=3.535534, giving normalized values \\[0.848528,1.131371\\]. Their mean is not zero. The core’s positive epsilon changes these numbers slightly and prevents division by zero.\n\nAssumptions Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified.\n\n## Engineering example\n\nSynthetic vector \\[1, 3\\] has mean 2 and population variance 1. With unit scale, zero shift and ε = 0.00001, LayerNorm gives approximately \\[−0.999995, 0.999995\\]. Separately, adding an update \\[0.5, −0.5\\] to the original vector gives \\[1.5, 2.5\\]; the two operations do different work.\n\nFor y=x+f(x), local derivative is 1+f′(x); if f′(x)=−0.2 the total is 0.8. For vector \\[3,4\\], RMS without epsilon is sqrt((9+16)/2)=sqrt12.5=3.535534, giving normalized values \\[0.848528,1.131371\\]. Their mean is not zero. The core’s positive epsilon changes these numbers slightly and prevents division by zero.\n\n## Distinctions and tradeoffs\n\nResidual streams carry current activations; they are not persistent personal memory.\n\nNormalization and residual placement affect stability. Low precision requires care with accumulation and numerical ranges.\n\nThese operations make very deep learned transformations practical.\n\n## Specifications and interfaces\n\nResidual add\n\nx + f(x)\n\nBoth operands must have compatible shapes.\n\nLayerNorm\n\n(x − μ) / √(σ² + ε)\n\nUsually includes learned affine parameters.\n\n### Interfaces\n\nEarlier layer Input\n\nActivation vector\n\nLater layer Output\n\nUpdated activation vector\n\n## Check your understanding\n\n### What travels through a residual connection?\n\n1.  An intermediate numerical representation\n2.  An archive of all training documents\n3.  Coolant\n\nAnswer and reasoning\n\nAn intermediate numerical representation\n\nIt carries activations through the computation graph.\n\n### Does RMS normalization force the output mean to zero?\n\n1.  Yes, for every input.\n2.  Only when residual paths are absent.\n3.  No; it rescales by root mean square without subtracting the mean.\n\nAnswer and reasoning\n\nNo; it rescales by root mean square without subtracting the mean.\n\nNo; it rescales by root mean square without subtracting the mean. Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified.\n\n## Historical context\n\n### 2016 · Layer normalization removes dependence on batch statistics\n\nBa, Kiros, and Hinton proposed normalization within a training example, with the same normalization computation at training and inference. This became useful for sequence models with varying batch conditions.\n\n[Ba et al. — Layer Normalization](https://arxiv.org/abs/1607.06450)\n\n## Evidence and scope\n\nThe example uses the linked LayerNorm definition over two features. RMSNorm, pre/post-normalization placement and learned parameters differ; the residual operands must have matching shapes, and neither operation guarantees stable training.\n\nMode: executable residual and RMS paths. Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [PyTorch — LayerNorm](https://docs.pytorch.org/docs/2.14/generated/torch.nn.LayerNorm.html)\n-   [He et al. — Deep Residual Learning](https://arxiv.org/abs/1512.03385)\n-   [Ba et al. — Layer Normalization](https://arxiv.org/abs/1607.06450)\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n-   [Root Mean Square Layer Normalization](https://arxiv.org/html/1910.07467v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/residual-normalization-89e7d8a2d9d3801e.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Feed-forward network](https://semiaifoundry.com/learn/atlas/feed-forward/)\n\n### Related lessons\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}