{"id":"reverse-mode-autodiff","name":"A differentiable computation graph","shortName":"A differentiable computation graph","role":"Accumulate every branch contribution to a reused parameter and check it by finite differences.","description":"A forward computation creates intermediate values. Reverse-mode differentiation traverses the dependency graph backward, multiplying each incoming derivative by the local derivative and adding contributions where paths meet. Reusing one parameter in several operations creates several paths to the loss; its final gradient is their sum. The optimizer acts only after these derivatives have been computed. This distinction lets a learner locate an incorrect derivative independently of an incorrect update rule. Numerical finite differences provide a separate local check, although their accuracy depends on step size and smoothness.","mechanism":["Save the forward operands required by local derivative rules.","Seed the scalar loss derivative with one.","Traverse operations in reverse dependency order and add, rather than overwrite, shared-input gradients.","Compare selected parameter derivatives with central differences while holding every other input fixed."],"learningObjective":"Accumulate every branch contribution to a reused parameter and check it by finite differences.","engineeringExample":"Let u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, [L(w+h)-L(w-h)]/(2h) is 32 in exact arithmetic for any nonzero h.","evidenceNotes":"Mode: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable core","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Accumulate every branch contribution to a reused parameter and check it by finite differences.","context":"Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Save the forward operands required by local derivative rules."},{"name":"Result and check","direction":"out","signal":"Accumulate every branch contribution to a reused parameter and check it by finite differences."}],"tradeoff":"Retaining intermediates reduces recomputation but consumes memory. Gradient checks are expensive if run on every parameter of a large model.","whyAI":"Join existing forward mechanisms into an initialized model with real gradients and measurable predictions. Accumulate every branch contribution to a reused parameter and check it by finite differences.","misconception":"It is a mistake to conclude: 24, because only the multiplication path is differentiable.","check":{"question":"What is the correct derivative with respect to w?","options":["24, because only the multiplication path is differentiable.","8, because the later path overwrites the earlier one.","32, because both computational paths contribute."],"answer":2,"explanation":"32, because both computational paths contribute. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters."},"prerequisites":["loss-optimizer","graph-tensor-ir","tensor-index-contracts"],"related":["loss-optimizer","graph-tensor-ir","tensor-index-contracts","tiny-causal-decoder"],"sources":[{"title":"PyTorch autograd mechanics","url":"https://docs.pytorch.org/docs/2.14/notes/autograd.html"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-base-model","subbranch":"A differentiable computation graph","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters.","science":[{"title":"A differentiable computation graph · worked relation","equation":"\\bar v_i\\mathrel{+}=\\bar v_j\\frac{\\partial v_j}{\\partial v_i},\\qquad \\frac{\\partial L}{\\partial w}\\approx\\frac{L(w+h)-L(w-h)}{2h}","explanation":"Let u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, [L(w+h)-L(w-h)]/(2h) is 32 in exact arithmetic for any nonzero h.","assumptions":"Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters."}],"history":[]}