semiAIfoundry

Generation · Atlas v1.3.0

Output head & decoding

Convert the final representation into the next token.

Explore in AtlasLogical structure · mapped onto physical hardware

Learning objectiveConvert a small logit vector to next-token probabilities and distinguish selection probability from truth.

The mechanism

The output projection assigns a logit to each candidate vocabulary token. A softmax can turn these scores into a probability distribution, after which a decoding rule selects an output. Greedy decoding takes a maximum; sampling introduces controlled randomness. The chosen ID is appended to the sequence and the process continues. A fluent output is therefore produced by many repeated computations, with behavior influenced by training, context, decoding, and any external tools.

The output head supplies logits for a probability distribution. Negative log likelihood compares that distribution with the observed target and differentiates to probability minus target indicator. Greedy decoding and categorical sampling consume the same probabilities differently. Neither updates the model’s weights by itself. The lifecycle generator retains a separate inference RNG so that requesting a sampled answer cannot change later training draws.

  1. Project the final hidden representation to vocabulary logits.
  2. Apply the decoding policy to select a token ID.
  3. Append that token and compute the next step.
  4. Apply stable log-softmax to all vocabulary logits.
  5. Gather the target probability and compute the logit derivative.
  6. Select greedy or seeded categorical output, keeping generation state separate from optimizer state.

Science and assumptions

A sequence probability factors into conditional predictions

p(x1,…,xT)=∏t=1Tp(xt∣x<t)p(xt=i∣x<t)=exp⁡(zi)∑jexp⁡(zj)\begin{aligned}p(x_1,\ldots,x_T)&=\prod_{t=1}^{T}p(x_t\mid x_{<t})\\p(x_t=i\mid x_{<t})&=\frac{\exp(z_i)}{\sum_j\exp(z_j)}\end{aligned}

Joint sequence probability factors into conditional next-token probabilities. At each step the probability of a vocabulary item is its exponentiated logit divided by the sum of exponentiated vocabulary logits.

Symbols and units
p
Joint or conditional model probability as indicated

probability

xₜ
Token at sequence position t

categorical token ID

x_<t
Tokens preceding position t

token-ID sequence

T
Sequence length

tokens

zᵢ
Vocabulary logit for token ID i at this step

dimensionless

i,j
Vocabulary-item indices

integer IDs

t
Sequence position

integer index

The chain rule factors a joint sequence distribution into conditional steps. Vocabulary logits z become normalized next-token probabilities. Greedy or sampling-based selection then turns the distribution into a particular sequence. A likely continuation can still assert something false, because probability under a language model is not a truth certificate.

Assumptions This is a standard autoregressive categorical decoder. It does not specify any model’s exposed sampling controls or internal scheduling; speculative decoding can verify several proposed tokens together while preserving a target distribution.

Likelihood, cross-entropy and sampled output · worked relation

pi=ezi∑jezj,L=−log⁡py,∂L∂zi=pi−1[i=y]p_i=\frac{e^{z_i}}{\sum_j e^{z_j}},\qquad L=-\log p_y,\qquad \frac{\partial L}{\partial z_i}=p_i-\mathbf1[i=y]

Connect logits, likelihood gradients and sampling without turning confidence into truth.

Symbols and units
z_i
Vocabulary logit

dimensionless

p_i
Next-token probability

dimensionless

y
Target class

index

L
One target negative log likelihood

nats

For logits [0,ln2,ln3], probabilities are [1/6,1/3,1/2]. With target class three, loss is ln2=0.693147 and gradient is [1/6,1/3,−1/2]. Greedy decoding chooses class three, while one categorical sample can choose another class. A sample different from the greedy class is not evidence that the probabilities or gradient were incorrectly computed.

Assumptions Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution.

Engineering example

Synthetic logits [0, ln 2, ln 3] exponentiate to [1, 2, 3], so softmax gives [1/6, 1/3, 1/2]. Greedy decoding chooses the third candidate; sampling can choose any of the three. A two-token sequence with conditional probabilities 1/2 then 1/4 has probability 1/8.

For logits [0,ln2,ln3], probabilities are [1/6,1/3,1/2]. With target class three, loss is ln2=0.693147 and gradient is [1/6,1/3,−1/2]. Greedy decoding chooses class three, while one categorical sample can choose another class. A sample different from the greedy class is not evidence that the probabilities or gradient were incorrectly computed.

Distinctions and tradeoffs

The output head chooses vocabulary symbols; factual verification requires additional evidence.

Sampling changes diversity and reproducibility. Output probability is not a calibrated guarantee that a statement is true.

Repeated decoding turns numerical transformations into an interactive response.

Specifications and interfaces

Logits
V scores

One score per vocabulary entry.

Autoregression
One dependent step after another

Batching and speculative methods alter execution scheduling.

Interfaces

Final block Input
Hidden representation
User / next step Output
Selected token ID and decoded text

Check your understanding

Does a high next-token probability establish factual truth?

  1. Yes
  2. No
  3. Only with more GPUs
Answer and reasoning

No

The distribution reflects the trained prediction task and context.

Why can a categorical sample differ from the greedy token?

  1. Sampling follows the whole distribution, including lower-probability outcomes.
  2. Sampling performs a hidden optimizer update.
  3. The largest logit is never used in sampling.
Answer and reasoning

Sampling follows the whole distribution, including lower-probability outcomes.

Sampling follows the whole distribution, including lower-probability outcomes. Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution.

Historical context

2020 · GPT-3 demonstrates broad in-context task conditioning

Brown and colleagues evaluated GPT-3 on many tasks using textual instructions and examples without task-specific gradient updates, illustrating the difference between conditioning generation and retraining weights.

Brown et al. — Language Models are Few-Shot Learners

Evidence and scope

This is the standard categorical autoregressive calculation already stated in the lesson. Toy logits are not model observations, and generation probability is not factual confidence or a claim about a private system’s decoding controls.

Mode: executable output probabilities and generation. Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.

Sources

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

Related lessons