{
  "schemaVersion": 1,
  "id": "atlas-lesson-output-head",
  "title": "Output head & decoding",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Convert the final representation into the next token.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/output-head/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Generation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This is the standard categorical autoregressive calculation already stated in the lesson. Toy logits are not model observations, and generation probability is not factual confidence or a claim about a private system’s decoding controls.\n\nMode: executable output probabilities and generation. Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/output-head-e7dbe11eabc20200.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#output-head",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "59fea3c10c2eb46341dd48e33010cce859281de0e0f2ffd2764d4ec014cbd33d",
  "contentLength": 8543,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-output-head.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-output-head.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Output head & decoding",
      "start": 27,
      "end": 352
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 352,
      "end": 1707
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1707,
      "end": 1735
    },
    {
      "id": "section-4",
      "title": "A sequence probability factors into conditional predictions",
      "start": 1735,
      "end": 3141
    },
    {
      "id": "section-5",
      "title": "Likelihood, cross-entropy and sampled output · worked relation",
      "start": 3141,
      "end": 4103
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4103,
      "end": 4750
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 4750,
      "end": 5082
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5082,
      "end": 5279
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5279,
      "end": 5398
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5398,
      "end": 5427
    },
    {
      "id": "section-11",
      "title": "Does a high next-token probability establish factual truth?",
      "start": 5427,
      "end": 5626
    },
    {
      "id": "section-12",
      "title": "Why can a categorical sample differ from the greedy token?",
      "start": 5626,
      "end": 6202
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 6202,
      "end": 6225
    },
    {
      "id": "section-14",
      "title": "2020 · GPT-3 demonstrates broad in-context task conditioning",
      "start": 6225,
      "end": 6597
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 6597,
      "end": 7418
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 7418,
      "end": 7783
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 7783,
      "end": 7806
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 7806,
      "end": 8076
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 8076,
      "end": 8543
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#output-head"
    },
    {
      "title": "Brown et al. — Language Models are Few-Shot Learners",
      "url": "https://arxiv.org/abs/2005.14165"
    },
    {
      "title": "Attention Is All You Need",
      "url": "https://arxiv.org/html/1706.03762v7"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/output-head-e7dbe11eabc20200.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Feed-forward network",
      "url": "https://semiaifoundry.com/learn/atlas/feed-forward/"
    },
    {
      "title": "Residual stream & normalization",
      "url": "https://semiaifoundry.com/learn/atlas/residual-normalization/"
    },
    {
      "title": "Tokenizer",
      "url": "https://semiaifoundry.com/learn/atlas/tokenizer/"
    },
    {
      "title": "Key–value cache",
      "url": "https://semiaifoundry.com/learn/atlas/kv-cache/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "f02083256c7f6a6a20f49528652beff8e65c87d524e111e39e665ecfb9c986d4",
  "contentFormat": "text/markdown",
  "content": "Generation · Atlas v1.3.0\n\n# Output head & decoding\n\nConvert the final representation into the next token.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#output-head)Logical structure · mapped onto physical hardware\n\nLearning objectiveConvert a small logit vector to next-token probabilities and distinguish selection probability from truth.\n\n## The mechanism\n\nThe output projection assigns a logit to each candidate vocabulary token. A softmax can turn these scores into a probability distribution, after which a decoding rule selects an output. Greedy decoding takes a maximum; sampling introduces controlled randomness. The chosen ID is appended to the sequence and the process continues. A fluent output is therefore produced by many repeated computations, with behavior influenced by training, context, decoding, and any external tools.\n\nThe output head supplies logits for a probability distribution. Negative log likelihood compares that distribution with the observed target and differentiates to probability minus target indicator. Greedy decoding and categorical sampling consume the same probabilities differently. Neither updates the model’s weights by itself. The lifecycle generator retains a separate inference RNG so that requesting a sampled answer cannot change later training draws.\n\n1.  Project the final hidden representation to vocabulary logits.\n2.  Apply the decoding policy to select a token ID.\n3.  Append that token and compute the next step.\n4.  Apply stable log-softmax to all vocabulary logits.\n5.  Gather the target probability and compute the logit derivative.\n6.  Select greedy or seeded categorical output, keeping generation state separate from optimizer state.\n\n## Science and assumptions\n\n### A sequence probability factors into conditional predictions\n\n$$\n\\begin{aligned}p(x_1,\\ldots,x_T)&=\\prod_{t=1}^{T}p(x_t\\mid x_{<t})\\\\p(x_t=i\\mid x_{<t})&=\\frac{\\exp(z_i)}{\\sum_j\\exp(z_j)}\\end{aligned}\n$$\n\nJoint sequence probability factors into conditional next-token probabilities. At each step the probability of a vocabulary item is its exponentiated logit divided by the sum of exponentiated vocabulary logits.\n\nSymbols and units\n\np\n\nJoint or conditional model probability as indicated\n\nprobability\n\nxₜ\n\nToken at sequence position t\n\ncategorical token ID\n\nx\\_<t\n\nTokens preceding position t\n\ntoken-ID sequence\n\nT\n\nSequence length\n\ntokens\n\nzᵢ\n\nVocabulary logit for token ID i at this step\n\ndimensionless\n\ni,j\n\nVocabulary-item indices\n\ninteger IDs\n\nt\n\nSequence position\n\ninteger index\n\nThe chain rule factors a joint sequence distribution into conditional steps. Vocabulary logits z become normalized next-token probabilities. Greedy or sampling-based selection then turns the distribution into a particular sequence. A likely continuation can still assert something false, because probability under a language model is not a truth certificate.\n\nAssumptions This is a standard autoregressive categorical decoder. It does not specify any model’s exposed sampling controls or internal scheduling; speculative decoding can verify several proposed tokens together while preserving a target distribution.\n\n### Likelihood, cross-entropy and sampled output · worked relation\n\n$$\np_i=\\frac{e^{z_i}}{\\sum_j e^{z_j}},\\qquad L=-\\log p_y,\\qquad \\frac{\\partial L}{\\partial z_i}=p_i-\\mathbf1[i=y]\n$$\n\nConnect logits, likelihood gradients and sampling without turning confidence into truth.\n\nSymbols and units\n\nz\\_i\n\nVocabulary logit\n\ndimensionless\n\np\\_i\n\nNext-token probability\n\ndimensionless\n\ny\n\nTarget class\n\nindex\n\nL\n\nOne target negative log likelihood\n\nnats\n\nFor logits \\[0,ln2,ln3\\], probabilities are \\[1/6,1/3,1/2\\]. With target class three, loss is ln2=0.693147 and gradient is \\[1/6,1/3,−1/2\\]. Greedy decoding chooses class three, while one categorical sample can choose another class. A sample different from the greedy class is not evidence that the probabilities or gradient were incorrectly computed.\n\nAssumptions Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution.\n\n## Engineering example\n\nSynthetic logits \\[0, ln 2, ln 3\\] exponentiate to \\[1, 2, 3\\], so softmax gives \\[1/6, 1/3, 1/2\\]. Greedy decoding chooses the third candidate; sampling can choose any of the three. A two-token sequence with conditional probabilities 1/2 then 1/4 has probability 1/8.\n\nFor logits \\[0,ln2,ln3\\], probabilities are \\[1/6,1/3,1/2\\]. With target class three, loss is ln2=0.693147 and gradient is \\[1/6,1/3,−1/2\\]. Greedy decoding chooses class three, while one categorical sample can choose another class. A sample different from the greedy class is not evidence that the probabilities or gradient were incorrectly computed.\n\n## Distinctions and tradeoffs\n\nThe output head chooses vocabulary symbols; factual verification requires additional evidence.\n\nSampling changes diversity and reproducibility. Output probability is not a calibrated guarantee that a statement is true.\n\nRepeated decoding turns numerical transformations into an interactive response.\n\n## Specifications and interfaces\n\nLogits\n\nV scores\n\nOne score per vocabulary entry.\n\nAutoregression\n\nOne dependent step after another\n\nBatching and speculative methods alter execution scheduling.\n\n### Interfaces\n\nFinal block Input\n\nHidden representation\n\nUser / next step Output\n\nSelected token ID and decoded text\n\n## Check your understanding\n\n### Does a high next-token probability establish factual truth?\n\n1.  Yes\n2.  No\n3.  Only with more GPUs\n\nAnswer and reasoning\n\nNo\n\nThe distribution reflects the trained prediction task and context.\n\n### Why can a categorical sample differ from the greedy token?\n\n1.  Sampling follows the whole distribution, including lower-probability outcomes.\n2.  Sampling performs a hidden optimizer update.\n3.  The largest logit is never used in sampling.\n\nAnswer and reasoning\n\nSampling follows the whole distribution, including lower-probability outcomes.\n\nSampling follows the whole distribution, including lower-probability outcomes. Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution.\n\n## Historical context\n\n### 2020 · GPT-3 demonstrates broad in-context task conditioning\n\nBrown and colleagues evaluated GPT-3 on many tasks using textual instructions and examples without task-specific gradient updates, illustrating the difference between conditioning generation and retraining weights.\n\n[Brown et al. — Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)\n\n## Evidence and scope\n\nThis is the standard categorical autoregressive calculation already stated in the lesson. Toy logits are not model observations, and generation probability is not factual confidence or a claim about a private system’s decoding controls.\n\nMode: executable output probabilities and generation. Finite logits, natural logarithms and one categorical target. Stable implementation subtracts maximum logit; that shift preserves the distribution. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [Brown et al. — Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165)\n-   [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/output-head-e7dbe11eabc20200.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Feed-forward network](https://semiaifoundry.com/learn/atlas/feed-forward/)\n-   [Residual stream & normalization](https://semiaifoundry.com/learn/atlas/residual-normalization/)\n\n### Related lessons\n\n-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n-   [Key–value cache](https://semiaifoundry.com/learn/atlas/kv-cache/)\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}