{
  "schemaVersion": 1,
  "id": "atlas-lesson-probability-conditioning-information",
  "title": "Probability, likelihood and stable information",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Connects model scores to probabilities, information and the gradients that guide learning.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/probability-conditioning-information/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "mathematical-foundations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/probability-conditioning-information-f8be7a0be8d8ae57.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#probability-conditioning-information",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "06d9e5460609cea06e06477b96a00ccb8fa4fb58b39d0cf780e9099fa0a39eb7",
  "contentLength": 6631,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-probability-conditioning-information.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-probability-conditioning-information.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Probability, likelihood and stable information",
      "start": 41,
      "end": 486
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 486,
      "end": 1714
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1714,
      "end": 1742
    },
    {
      "id": "section-4",
      "title": "Probability, likelihood and stable information",
      "start": 1742,
      "end": 2842
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2842,
      "end": 3195
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3195,
      "end": 3713
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3713,
      "end": 4400
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4400,
      "end": 4705
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4705,
      "end": 4734
    },
    {
      "id": "section-10",
      "title": "What does adding the same finite constant to all logits do mathematically?",
      "start": 4734,
      "end": 5297
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 5297,
      "end": 5853
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5853,
      "end": 6331
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 6331,
      "end": 6354
    },
    {
      "id": "section-14",
      "title": "Related lessons",
      "start": 6354,
      "end": 6631
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#probability-conditioning-information"
    },
    {
      "title": "Goodfellow, Bengio and Courville · Probability and information theory",
      "url": "https://www.deeplearningbook.org/contents/prob.html"
    },
    {
      "title": "Goodfellow, Bengio and Courville · Numerical computation",
      "url": "https://www.deeplearningbook.org/contents/numerical.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/probability-conditioning-information-f8be7a0be8d8ae57.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    }
  ],
  "recordHash": "7bc1301bbc141f860bece6b7c0945bbc7a574574af1f46c91ed550a0c3ca9758",
  "contentFormat": "text/markdown",
  "content": "mathematical-foundations · Atlas v1.3.0\n\n# Probability, likelihood and stable information\n\nConnects model scores to probabilities, information and the gradients that guide learning.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#probability-conditioning-information)Mathematical and data space; dimensions are not physical distance\n\nLearning objectiveConvert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either.\n\n## The mechanism\n\nA logit is an unnormalized score. Softmax turns a finite score vector into probabilities by exponentiating and dividing by the total. Negative log likelihood measures how much probability was assigned to the observed target; it is not a correctness score for a generated sentence. Computing exponentials directly can overflow even when the intended probabilities are ordinary numbers. Subtracting the largest logit preserves all score differences and therefore preserves the distribution. The log-sum-exp form calculates loss without first rounding a tiny target probability to zero. The workbench shows both the stable result and the failure of the naive exponential sum, then exposes the derivative with respect to each score.\n\n1.  Declare the outcome set and observed target. Scores alone need not sum to one and can be negative.\n2.  Subtract the largest logit before exponentiating, sum shifted exponentials and normalize them.\n3.  Calculate target loss in log space and entropy over the resulting distribution, using natural logarithms in this example.\n4.  Inspect the loss derivative pᵢ−1\\[i=y\\]. The target component differs from the other components because the observed outcome supplies supervision.\n\n## Science and assumptions\n\n### Probability, likelihood and stable information\n\n$$\np_i=\\frac{e^{z_i-m}}{\\sum_j e^{z_j-m}},\\quad L=\\log\\sum_j e^{z_j-m}+m-z_y,\\quad \\frac{\\partial L}{\\partial z_i}=p_i-\\mathbf 1[i=y]\n$$\n\nSubtract the maximum score m, normalize shifted exponentials, compute target log loss, and differentiate to obtain probability minus target indicator.\n\nSymbols and units\n\nzᵢ,m\n\nLogit and maximum logit\n\ndimensionless\n\npᵢ\n\nPredicted outcome probability\n\nprobability\n\ny\n\nObserved target index\n\nindex\n\nL\n\nNegative log likelihood\n\nnats\n\nScores \\[0,ln 2,ln 3\\] normalize to \\[1/6,1/3,1/2\\]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.\n\nAssumptions Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.\n\n## Engineering example\n\nScores \\[0,ln 2,ln 3\\] normalize to \\[1/6,1/3,1/2\\]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.\n\n## Distinctions and tradeoffs\n\nHigh assigned probability is a statement made by the model. It is not proof that an answer is true or that the model is calibrated.\n\nA probability distribution enables likelihood-based optimization and uncertainty analysis, but calibration and task correctness require empirical evaluation on identified data.\n\nLanguage-model output heads and cross-entropy training repeatedly perform this operation. Stable arithmetic is part of preserving the mathematical objective on real machines.\n\n## Specifications and interfaces\n\nTeaching specimen\n\nProbability and loss\n\nScores \\[0,ln 2,ln 3\\] normalize to \\[1/6,1/3,1/2\\]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.\n\nEvidence boundary\n\nExplicit computed example\n\nFinite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.\n\n### Interfaces\n\nInputs and convention Input\n\nDeclare the outcome set and observed target. Scores alone need not sum to one and can be negative.\n\nComputed result and scope Output\n\nConvert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either.\n\n## Check your understanding\n\n### What does adding the same finite constant to all logits do mathematically?\n\n1.  It necessarily makes the model more confident\n2.  It changes the target label\n3.  It leaves the softmax distribution unchanged\n\nAnswer and reasoning\n\nIt leaves the softmax distribution unchanged\n\nIt leaves the softmax distribution unchanged. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.\n\n## Evidence and scope\n\nThe workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.\n\nOpen Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.\n\n## Sources\n\n-   [Goodfellow, Bengio and Courville · Probability and information theory](https://www.deeplearningbook.org/contents/prob.html)\n-   [Goodfellow, Bengio and Courville · Numerical computation](https://www.deeplearningbook.org/contents/numerical.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/probability-conditioning-information-f8be7a0be8d8ae57.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Related lessons\n\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)"
}