semiAIfoundry

mathematical-foundations · Atlas v1.3.0

Probability, likelihood and stable information

Connects model scores to probabilities, information and the gradients that guide learning.

Explore in AtlasMathematical and data space; dimensions are not physical distance

Learning objectiveConvert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either.

The mechanism

A logit is an unnormalized score. Softmax turns a finite score vector into probabilities by exponentiating and dividing by the total. Negative log likelihood measures how much probability was assigned to the observed target; it is not a correctness score for a generated sentence. Computing exponentials directly can overflow even when the intended probabilities are ordinary numbers. Subtracting the largest logit preserves all score differences and therefore preserves the distribution. The log-sum-exp form calculates loss without first rounding a tiny target probability to zero. The workbench shows both the stable result and the failure of the naive exponential sum, then exposes the derivative with respect to each score.

  1. Declare the outcome set and observed target. Scores alone need not sum to one and can be negative.
  2. Subtract the largest logit before exponentiating, sum shifted exponentials and normalize them.
  3. Calculate target loss in log space and entropy over the resulting distribution, using natural logarithms in this example.
  4. Inspect the loss derivative pᵢ−1[i=y]. The target component differs from the other components because the observed outcome supplies supervision.

Science and assumptions

Probability, likelihood and stable information

pi=ezi−m∑jezj−m,L=log⁡∑jezj−m+m−zy,∂L∂zi=pi−1[i=y]p_i=\frac{e^{z_i-m}}{\sum_j e^{z_j-m}},\quad L=\log\sum_j e^{z_j-m}+m-z_y,\quad \frac{\partial L}{\partial z_i}=p_i-\mathbf 1[i=y]

Subtract the maximum score m, normalize shifted exponentials, compute target log loss, and differentiate to obtain probability minus target indicator.

Symbols and units
zᵢ,m
Logit and maximum logit

dimensionless

pᵢ
Predicted outcome probability

probability

y
Observed target index

index

L
Negative log likelihood

nats

Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.

Assumptions Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.

Engineering example

Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.

Distinctions and tradeoffs

High assigned probability is a statement made by the model. It is not proof that an answer is true or that the model is calibrated.

A probability distribution enables likelihood-based optimization and uncertainty analysis, but calibration and task correctness require empirical evaluation on identified data.

Language-model output heads and cross-entropy training repeatedly perform this operation. Stable arithmetic is part of preserving the mathematical objective on real machines.

Specifications and interfaces

Teaching specimen
Probability and loss

Scores [0,ln 2,ln 3] normalize to [1/6,1/3,1/2]. If target class is 2, loss is ln 2 ≈0.6931 nats. Adding 1000 to every score leaves the mathematical probabilities and loss unchanged, while naive exponentials overflow in ordinary floating arithmetic. Finite-difference checks independently verify the three loss derivatives.

Evidence boundary
Explicit computed example

Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.

Interfaces

Inputs and convention Input
Declare the outcome set and observed target. Scores alone need not sum to one and can be negative.
Computed result and scope Output
Convert logits to a normalized distribution, calculate target loss and explain why a common offset should not change either.

Check your understanding

What does adding the same finite constant to all logits do mathematically?

  1. It necessarily makes the model more confident
  2. It changes the target label
  3. It leaves the softmax distribution unchanged
Answer and reasoning

It leaves the softmax distribution unchanged

It leaves the softmax distribution unchanged. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.

Evidence and scope

The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Finite logits, categorical outcomes and a single observed class. Natural logarithms are used. Conditioning on further evidence requires its own likelihood model; the workbench changes scores rather than demonstrating Bayesian updating.

Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.

Sources

Published lesson data · Atlas releases and licensing

Continue exploring

Related lessons