{
  "schemaVersion": 1,
  "id": "atlas-lesson-reverse-mode-autodiff",
  "title": "A differentiable computation graph",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Accumulate every branch contribution to a reused parameter and check it by finite differences.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-base-model",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/reverse-mode-autodiff-4d49f015b5c53d0a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#reverse-mode-autodiff",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "aab0a3a4f3388c283977ca4d4be6b3bb27cc1f17f99ce5fc191ca40c269c9f8e",
  "contentLength": 6032,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-reverse-mode-autodiff.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-reverse-mode-autodiff.md",
  "sections": [
    {
      "id": "section-1",
      "title": "A differentiable computation graph",
      "start": 43,
      "end": 453
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 453,
      "end": 1433
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1433,
      "end": 1461
    },
    {
      "id": "section-4",
      "title": "A differentiable computation graph · worked relation",
      "start": 1461,
      "end": 2552
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2552,
      "end": 2909
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2909,
      "end": 3375
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3375,
      "end": 3828
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3828,
      "end": 4052
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4052,
      "end": 4081
    },
    {
      "id": "section-10",
      "title": "What is the correct derivative with respect to w?",
      "start": 4081,
      "end": 4607
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4607,
      "end": 5246
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5246,
      "end": 5548
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5548,
      "end": 5571
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5571,
      "end": 5900
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5900,
      "end": 6032
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#reverse-mode-autodiff"
    },
    {
      "title": "PyTorch autograd mechanics",
      "url": "https://docs.pytorch.org/docs/2.14/notes/autograd.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/reverse-mode-autodiff-4d49f015b5c53d0a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Model graph and tensor intermediate representation",
      "url": "https://semiaifoundry.com/learn/atlas/graph-tensor-ir/"
    },
    {
      "title": "Tensor shapes, indices and contractions",
      "url": "https://semiaifoundry.com/learn/atlas/tensor-index-contracts/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "b1b858f8de7b6557934158d20eecaf82a5ec920acfd8c299133fd6dd90e714f9",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-base-model · Atlas v1.3.0\n\n# A differentiable computation graph\n\nAccumulate every branch contribution to a reused parameter and check it by finite differences.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#reverse-mode-autodiff)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveAccumulate every branch contribution to a reused parameter and check it by finite differences.\n\n## The mechanism\n\nA forward computation creates intermediate values. Reverse-mode differentiation traverses the dependency graph backward, multiplying each incoming derivative by the local derivative and adding contributions where paths meet. Reusing one parameter in several operations creates several paths to the loss; its final gradient is their sum. The optimizer acts only after these derivatives have been computed. This distinction lets a learner locate an incorrect derivative independently of an incorrect update rule. Numerical finite differences provide a separate local check, although their accuracy depends on step size and smoothness.\n\n1.  Save the forward operands required by local derivative rules.\n2.  Seed the scalar loss derivative with one.\n3.  Traverse operations in reverse dependency order and add, rather than overwrite, shared-input gradients.\n4.  Compare selected parameter derivatives with central differences while holding every other input fixed.\n\n## Science and assumptions\n\n### A differentiable computation graph · worked relation\n\n$$\n\\bar v_i\\mathrel{+}=\\bar v_j\\frac{\\partial v_j}{\\partial v_i},\\qquad \\frac{\\partial L}{\\partial w}\\approx\\frac{L(w+h)-L(w-h)}{2h}\n$$\n\nAccumulate every branch contribution to a reused parameter and check it by finite differences.\n\nSymbols and units\n\nv\\_i,v\\_j\n\nIntermediate scalar values\n\ndimensionless\n\nbar v\\_i\n\nAccumulated derivative of loss with respect to v\\_i\n\nloss per value\n\nw\n\nReused trainable scalar\n\ndimensionless\n\nh\n\nFinite-difference perturbation\n\ndimensionless\n\nL\n\nScalar objective\n\ndimensionless\n\nLet u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, \\[L(w+h)-L(w-h)\\]/(2h) is 32 in exact arithmetic for any nonzero h.\n\nAssumptions Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters.\n\n## Engineering example\n\nLet u=wx and L=(u+w)^2/2. At w=2 and x=3, u=6, u+w=8 and L=32. The path through u contributes 8×3=24 to dL/dw; the direct w path contributes 8, giving 32. Treating the second path as a replacement would give 8, while omitting it would give 24. For this quadratic, \\[L(w+h)-L(w-h)\\]/(2h) is 32 in exact arithmetic for any nonzero h.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: 24, because only the multiplication path is differentiable.\n\nRetaining intermediates reduces recomputation but consumes memory. Gradient checks are expensive if run on every parameter of a large model.\n\nJoin existing forward mechanisms into an initialized model with real gradients and measurable predictions. Accumulate every branch contribution to a reused parameter and check it by finite differences.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable core\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nAccumulate every branch contribution to a reused parameter and check it by finite differences.\n\nSmooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters.\n\n### Interfaces\n\nInputs and state Input\n\nSave the forward operands required by local derivative rules.\n\nResult and check Output\n\nAccumulate every branch contribution to a reused parameter and check it by finite differences.\n\n## Check your understanding\n\n### What is the correct derivative with respect to w?\n\n1.  24, because only the multiplication path is differentiable.\n2.  8, because the later path overwrites the earlier one.\n3.  32, because both computational paths contribute.\n\nAnswer and reasoning\n\n32, because both computational paths contribute.\n\n32, because both computational paths contribute. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters.\n\n## Evidence and scope\n\nMode: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable core. Smooth scalar graph and nonzero finite-difference step; floating-point cancellation can limit very small h. Lifecycle tape executes these mechanisms for all 423 parameters.\n\n## Sources\n\n-   [PyTorch autograd mechanics](https://docs.pytorch.org/docs/2.14/notes/autograd.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/reverse-mode-autodiff-4d49f015b5c53d0a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Model graph and tensor intermediate representation](https://semiaifoundry.com/learn/atlas/graph-tensor-ir/)\n-   [Tensor shapes, indices and contractions](https://semiaifoundry.com/learn/atlas/tensor-index-contracts/)\n\n### Related lessons\n\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}