{"id":"probability-conditioning-information","name":"Probability, likelihood and stable information","shortName":"Probability and loss","role":"Connects model scores to probabilities, information and the gradients that guide learning.","description":"A logit is an unnormalized score. Softmax turns a finite score vector into probabilities by exponentiating and dividing by the total. Negative log likelihood measures how much probability was assigned to the observed target; it is not a correctness score for a generated sentence. Computing exponentials directly can overflow even when the intended probabilities are ordinary numbers. Subtracting the largest logit preserves all score differences and therefore preserves the distribution. The log-sum-exp form calculates loss without first rounding a tiny target probability to zero. The workbench shows both the stable result and the failure of the naive exponential sum, then exposes the derivative with respect to each score.","mechanism":["Declare the outcome set and observed target. Scores alone need not sum to one and can be negative.","Subtract the largest logit before exponentiating, sum shifted exponentials and normalize them.","Calculate target loss in log space and entropy over the resulting distribution, using natural logarithms in this example.","Inspect the loss derivative pᵢ−1[i=y]. The target component differs from the other components because the observed outcome supplies supervision."],"learningObjective":"Convert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either.","engineeringExample":"Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.","evidenceNotes":"The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.","specs":[{"label":"Teaching specimen","value":"Probability and loss","context":"Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives."},{"label":"Evidence boundary","value":"Explicit computed example","context":"Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating."}],"interfaces":[{"name":"Inputs and convention","direction":"in","signal":"Declare the outcome set and observed target. Scores alone need not sum to one and can be negative."},{"name":"Computed result and scope","direction":"out","signal":"Convert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either."}],"tradeoff":"A probability distribution enables likelihood-based optimization and uncertainty analysis, but calibration and task correctness require empirical evaluation on identified data.","whyAI":"Language-model output heads and cross-entropy training repeatedly perform this operation. Stable arithmetic is part of preserving the mathematical objective on real machines.","misconception":"High assigned probability is a statement made by the model. It is not proof that an answer is true or that the model is calibrated.","check":{"question":"What does adding the same finite constant to all logits do mathematically?","options":["It necessarily makes the model more confident","It changes the target label","It leaves the softmax distribution unchanged"],"answer":2,"explanation":"It leaves the softmax distribution unchanged. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating."},"prerequisites":[],"related":["output-head","loss-optimizer","evaluation"],"sources":[{"title":"Goodfellow, Bengio and Courville · Probability and information theory","url":"https://www.deeplearningbook.org/contents/prob.html"},{"title":"Goodfellow, Bengio and Courville · Numerical computation","url":"https://www.deeplearningbook.org/contents/numerical.html"}],"level":"model","scale":"Mathematical and data space; dimensions are not physical distance","kind":"model","branch":"mathematical-foundations","subbranch":"Probability and loss","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.","science":[{"title":"Probability, likelihood and stable information","equation":"Subtract the maximum score m, normalize shifted exponentials, compute target log loss, and differentiate to obtain probability minus target indicator.","explanation":"Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.","assumptions":"Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating."}]}