# A differentiable computation graph Canonical: https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-base-model · Atlas v1.3.0 # A differentiable computation graph Accumulate every branch contribution to a reused parameter and check it by finite differences. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#reverse-mode-autodiff)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveAccumulate every branch contribution to a reused parameter and check it by finite differences. ## The mechanism A forward computation creates intermediate values. Reverse-mode differentiation traverses the dependency graph backward, multiplying each incoming derivative by the local derivative and adding contributions where paths meet. Reusing one parameter in several operations creates several paths to the loss; its final gradient is their sum. The optimizer acts only after these derivatives have been computed. This distinction lets a learner locate an incorrect derivative independently of an incorrect update rule. Numerical finite differences provide a separate local check, although their accuracy depends on step size and smoothness. 1. Save the forward operands required by local derivative rules. 2. Seed the scalar loss derivative with one. 3. Traverse operations in reverse dependency order and add, rather than overwrite, shared-input gradients. 4. Compare selected parameter derivatives with central differences while holding every other input fixed. ## Science and assumptions ### A differentiable computation graph · worked relation $$ \bar v_i\mathrel{+}=\bar v_j\frac{\partial v_j}{\partial v_i},\qquad \frac{\partial L}{\partial w}\approx\frac{L(w+h)-L(w-h)}{2h} $$ Accumulate every branch contribution to a reused parameter and check it by finite differences. Symbols and units v\_i,v\_j Intermediate scalar values dimensionless bar v\_i Accumulated derivative of loss with respect to v\_i loss per value w Reused trainable scalar dimensionless h Finite-difference perturbation dimensionless L Scalar objective dimensionless Let u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, \[L(w+h)-L(w-h)\]/(2h) is 32 in exact arithmetic for any nonzero h. Assumptions Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. ## Engineering example Let u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, \[L(w+h)-L(w-h)\]/(2h) is 32 in exact arithmetic for any nonzero h. ## Distinctions and tradeoffs It is a mistake to conclude: 24, because only the multiplication path is differentiable. Retaining intermediates reduces recomputation but consumes memory. Gradient checks are expensive if run on every parameter of a large model. Join existing forward mechanisms into an initialized model with real gradients and measurable predictions. Accumulate every branch contribution to a reused parameter and check it by finite differences. ## Specifications and interfaces Learning mode executable core The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Accumulate every branch contribution to a reused parameter and check it by finite differences. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. ### Interfaces Inputs and state Input Save the forward operands required by local derivative rules. Result and check Output Accumulate every branch contribution to a reused parameter and check it by finite differences. ## Check your understanding ### What is the correct derivative with respect to w? 1. 24, because only the multiplication path is differentiable. 2. 8, because the later path overwrites the earlier one. 3. 32, because both computational paths contribute. Answer and reasoning 32, because both computational paths contribute. 32, because both computational paths contribute. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. ## Evidence and scope Mode: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. ## Sources - [PyTorch autograd mechanics](https://docs.pytorch.org/docs/2.14/notes/autograd.html) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/reverse-mode-autodiff-4d49f015b5c53d0a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Model graph and tensor intermediate representation](https://semiaifoundry.com/learn/atlas/graph-tensor-ir/) - [Tensor shapes, indices and contractions](https://semiaifoundry.com/learn/atlas/tensor-index-contracts/) ### Related lessons - [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)