{"id":"residual-normalization","level":"model","name":"Residual stream & normalization","shortName":"Residual stream & normalization","role":"Keep a deep sequence of transformations trainable.","description":"Residual connections add a block’s output to a continuing representation. This lets each block contribute an update and provides a direct route through a deep computation graph. Normalization controls aspects of numerical scale; LayerNorm uses the mean and variance over specified feature dimensions, while other normalizers differ. Placement before or after sublayers matters. Neither operation creates persistent autobiographical memory: the stream is an intermediate activation for the current computation.\n\nA residual addition passes an input directly to the output alongside a learned transformation, creating an identity term in the derivative. Normalization modifies feature scale before or after a sublayer according to the declared architecture. RMS normalization does not subtract the mean, and its learned gain and epsilon conventions must be stated. The lifecycle variant uses pre-RMS normalization with fixed gain one and epsilon 1e-5; it is not exactly the original Transformer normalization placement.","mechanism":["Normalize the features at the architecture’s chosen location.","Compute the block’s transformation.","Add the compatible result to the continuing representation.","Trace the identity branch separately from the sublayer branch.","Compute mean square over the declared feature axis, with epsilon placed explicitly.","Compare actual activation and gradient values rather than assuming a residual guarantees stable learning."],"specs":[{"label":"Residual add","value":"x + f(x)","context":"Both operands must have compatible shapes."},{"label":"LayerNorm","value":"(x − μ) / √(σ² + ε)","context":"Usually includes learned affine parameters."}],"interfaces":[{"name":"Earlier layer","direction":"in","signal":"Activation vector"},{"name":"Later layer","direction":"out","signal":"Updated activation vector"}],"tradeoff":"Normalization and residual placement affect stability. Low precision requires care with accumulation and numerical ranges.","whyAI":"These operations make very deep learned transformations practical.","related":["attention","feed-forward","loss-optimizer","reverse-mode-autodiff","tiny-causal-decoder"],"sources":[{"title":"PyTorch — LayerNorm","url":"https://docs.pytorch.org/docs/2.14/generated/torch.nn.LayerNorm.html"},{"title":"He et al. — Deep Residual Learning","url":"https://arxiv.org/abs/1512.03385"},{"title":"Ba et al. — Layer Normalization","url":"https://arxiv.org/abs/1607.06450"},{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"},{"title":"Root Mean Square Layer Normalization","url":"https://arxiv.org/html/1910.07467v1"}],"scale":"Logical structure · mapped onto physical hardware","kind":"residual-normalization","misconception":"Residual streams carry current activations; they are not persistent personal memory.","check":{"question":"What travels through a residual connection?","options":["An intermediate numerical representation","An archive of all training documents","Coolant"],"answer":0,"explanation":"It carries activations through the computation graph."},"branch":"Transformer computation","subbranch":"Depth and numerical stability","science":[{"title":"Normalization and a direct gradient route","equation":"LN(x) = γ ⊙ (x−μ)/√(σ²+ε) + β; y = x + F(x); ∂y/∂x = I + J_F(x)","explanation":"LayerNorm measures the mean and variance of one representation across its normalized features, then applies learned scale and shift. Separately, a residual add supplies an identity term in the Jacobian, giving gradients a direct path alongside the learned transformation. Neither guarantees that all deep networks train stably.","assumptions":"The residual equation describes an identity skip with matching shapes; F may include normalization. LayerNorm uses population variance over the chosen features. RMSNorm omits mean subtraction, and pre-norm versus post-norm changes the full block Jacobian."},{"title":"Activation scale and gradient flow · worked relation","equation":"\\frac{dy}{dx}=1+f^{\\prime}(x),\\qquad \\operatorname{RMSNorm}(x)_i=\\frac{x_i}{\\sqrt{d^{-1}\\sum_j x_j^2+\\epsilon}}","explanation":"For y=x+f(x), local derivative is 1+f′(x); if f′(x)=−0.2 the total is 0.8. For vector [3,4], RMS without epsilon is sqrt((9+16)/2)=sqrt12.5=3.535534, giving normalized values [0.848528,1.131371]. Their mean is not zero. The core’s positive epsilon changes these numbers slightly and prevents division by zero.","assumptions":"Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified."}],"history":[{"year":2016,"title":"Layer normalization removes dependence on batch statistics","significance":"Ba, Kiros, and Hinton proposed normalization within a training example, with the same normalization computation at training and inference. This became useful for sequence models with varying batch conditions.","source":{"title":"Ba et al. — Layer Normalization","url":"https://arxiv.org/abs/1607.06450"}}],"prerequisites":["embedding","feed-forward"],"learningObjective":"Compute a small LayerNorm example and distinguish normalization from a residual addition.","engineeringExample":"Synthetic vector [1, 3] has mean 2 and population variance 1. With unit scale, zero shift and ε = 0.00001, LayerNorm gives approximately [−0.999995, 0.999995]. Separately, adding an update [0.5, −0.5] to the original vector gives [1.5, 2.5]; the two operations do different work.\n\nFor y=x+f(x), local derivative is 1+f′(x); if f′(x)=−0.2 the total is 0.8. For vector [3,4], RMS without epsilon is sqrt((9+16)/2)=sqrt12.5=3.535534, giving normalized values [0.848528,1.131371]. Their mean is not zero. The core’s positive epsilon changes these numbers slightly and prevents division by zero.","evidenceNotes":"The example uses the linked LayerNorm definition over two features. RMSNorm, pre/post-normalization placement and learned parameters differ; the residual operands must have matching shapes, and neither operation guarantees stable training.\n\nMode: executable residual and RMS paths. Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","checks":[{"question":"Does RMS normalization force the output mean to zero?","options":["Yes, for every input.","Only when residual paths are absent.","No; it rescales by root mean square without subtracting the mean."],"answer":2,"explanation":"No; it rescales by root mean square without subtracting the mean. Fixed gain one; epsilon zero only for the explicitly labeled hand calculation. Core epsilon is 1e-5, so exact scale invariance is qualified."}],"labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}