{
  "schemaVersion": 1,
  "id": "atlas-lesson-causal-pretraining-objective",
  "title": "Next-token supervision from raw sequences",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-pretraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/causal-pretraining-objective-0045cc6f8ca8a55b.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#causal-pretraining-objective",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "2124b0d54fd316ec7657c90f8237469c28c2136852ce186edad9424cb1d0d7d5",
  "contentLength": 6013,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-causal-pretraining-objective.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-causal-pretraining-objective.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Next-token supervision from raw sequences",
      "start": 44,
      "end": 480
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 480,
      "end": 1387
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1387,
      "end": 1415
    },
    {
      "id": "section-4",
      "title": "Next-token supervision from raw sequences · worked relation",
      "start": 1415,
      "end": 2455
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2455,
      "end": 2827
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2827,
      "end": 3263
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3263,
      "end": 3700
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3700,
      "end": 3929
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3929,
      "end": 3958
    },
    {
      "id": "section-10",
      "title": "Which comparison uses the same next-token objective?",
      "start": 3958,
      "end": 4518
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4518,
      "end": 5113
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5113,
      "end": 5462
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5462,
      "end": 5485
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5485,
      "end": 5740
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5740,
      "end": 6013
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#causal-pretraining-objective"
    },
    {
      "title": "2 OLMo 2 Furious",
      "url": "https://arxiv.org/html/2501.00656v1"
    },
    {
      "title": "Attention Is All You Need",
      "url": "https://arxiv.org/html/1706.03762v7"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/causal-pretraining-objective-0045cc6f8ca8a55b.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Initialization, learning rate and stable updates",
      "url": "https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/"
    },
    {
      "title": "Gradient accumulation under a fixed global objective",
      "url": "https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/"
    }
  ],
  "recordHash": "2c2dc50a548da67ac2e7c0f3d546a2d7c61d7bfe51191873411396f27f2bef16",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-pretraining · Atlas v1.3.0\n\n# Next-token supervision from raw sequences\n\nConstruct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#causal-pretraining-objective)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveConstruct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\n## The mechanism\n\nAutoregressive pretraining asks a model to predict each next token from earlier inputs. The target is the shifted sequence, not the token already visible at the same position. Summing negative log probabilities gives a sequence score; averaging included targets gives the training objective. Exponentiating this mean yields perplexity under the same tokenization and mask. Answer accuracy is a different measurement: it observes selected decisions rather than every token probability. A model can become better at punctuation or repeated format markers while its answer decisions remain unchanged.\n\n1.  Pair input position t with token t+1 and mark valid targets.\n2.  Compute stable log-softmax and gather each target’s log probability.\n3.  Sum losses and divide by included targets across the batch.\n4.  Fit a simple baseline on training data only and compare the same reserved targets.\n\n## Science and assumptions\n\n### Next-token supervision from raw sequences · worked relation\n\n$$\nL=-\\frac{1}{M}\\sum_{t:m_t=1}\\log p_\\theta(x_{t+1}\\mid x_{\\le t}),\\qquad \\operatorname{PPL}=e^L\n$$\n\nConstruct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\nSymbols and units\n\nL\n\nMean included next-token NLL\n\nnats per token\n\nM\n\nIncluded target count\n\ntokens\n\nm\\_t\n\nTarget inclusion flag\n\ndimensionless\n\np\\_theta\n\nConditional target probability\n\ndimensionless\n\nPPL\n\nPerplexity under this tokenization\n\ndimensionless\n\nFor three target probabilities \\[1/2,1/4,1/2\\], total NLL is ln2+ln4+ln2=ln16=2.772589 nats. Mean NLL is 0.924196 nats and perplexity is 16^(1/3)=2.519842. A uniform nine-token predictor has mean NLL ln9=2.197225 and perplexity 9. Feeding the answer token at the very position being scored would invalidate this comparison through target leakage.\n\nAssumptions Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.\n\n## Engineering example\n\nFor three target probabilities \\[1/2,1/4,1/2\\], total NLL is ln2+ln4+ln2=ln16=2.772589 nats. Mean NLL is 0.924196 nats and perplexity is 16^(1/3)=2.519842. A uniform nine-token predictor has mean NLL ln9=2.197225 and perplexity 9. Feeding the answer token at the very position being scored would invalidate this comparison through target leakage.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Model answer accuracy versus baseline perplexity.\n\nToken likelihood supplies dense supervision but weights frequent formatting and rare task-critical targets according to their actual counts.\n\nTurn an objective into a stable, restartable experiment with controlled budgets. Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable core\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nConstruct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\nNatural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.\n\n### Interfaces\n\nInputs and state Input\n\nPair input position t with token t+1 and mark valid targets.\n\nResult and check Output\n\nConstruct shifted next-token targets and interpret token likelihood separately from answer accuracy.\n\n## Check your understanding\n\n### Which comparison uses the same next-token objective?\n\n1.  Training loss after showing the target at its own input position.\n2.  Model and baseline scored on identical shifted targets and masks.\n3.  Model answer accuracy versus baseline perplexity.\n\nAnswer and reasoning\n\nModel and baseline scored on identical shifted targets and masks.\n\nModel and baseline scored on identical shifted targets and masks. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.\n\n## Evidence and scope\n\nMode: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.\n\n## Sources\n\n-   [2 OLMo 2 Furious](https://arxiv.org/html/2501.00656v1)\n-   [Attention Is All You Need](https://arxiv.org/html/1706.03762v7)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/causal-pretraining-objective-0045cc6f8ca8a55b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n\n### Related lessons\n\n-   [Initialization, learning rate and stable updates](https://semiaifoundry.com/learn/atlas/optimizer-schedule-stability/)\n-   [Gradient accumulation under a fixed global objective](https://semiaifoundry.com/learn/atlas/microbatch-equivalent-updates/)"
}