{"id":"causal-pretraining-objective","name":"Next-token supervision from raw sequences","shortName":"Next-token supervision from raw sequences","role":"Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.","description":"Autoregressive pretraining asks a model to predict each next token from earlier inputs. The target is the shifted sequence, not the token already visible at the same position. Summing negative log probabilities gives a sequence score; averaging included targets gives the training objective. Exponentiating this mean yields perplexity under the same tokenization and mask. Answer accuracy is a different measurement: it observes selected decisions rather than every token probability. A model can become better at punctuation or repeated format markers while its answer decisions remain unchanged.","mechanism":["Pair input position t with token t+1 and mark valid targets.","Compute stable log-softmax and gather each target’s log probability.","Sum losses and divide by included targets across the batch.","Fit a simple baseline on training data only and compare the same reserved targets."],"learningObjective":"Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.","engineeringExample":"For three target probabilities [1/2,1/4,1/2], total NLL is ln2+ln4+ln2=ln16=2.772589 nats. Mean NLL is 0.924196 nats and perplexity is 16^(1/3)=2.519842. A uniform nine-token predictor has mean NLL ln9=2.197225 and perplexity 9. Feeding the answer token at the very position being scored would invalidate this comparison through target leakage.","evidenceNotes":"Mode: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable core","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.","context":"Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Pair input position t with token t+1 and mark valid targets."},{"name":"Result and check","direction":"out","signal":"Construct shifted next-token targets and interpret token likelihood separately from answer accuracy."}],"tradeoff":"Token likelihood supplies dense supervision but weights frequent formatting and rare task-critical targets according to their actual counts.","whyAI":"Turn an objective into a stable, restartable experiment with controlled budgets. Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.","misconception":"It is a mistake to conclude: Model answer accuracy versus baseline perplexity.","check":{"question":"Which comparison uses the same next-token objective?","options":["Training loss after showing the target at its own input position.","Model and baseline scored on identical shifted targets and masks.","Model answer accuracy versus baseline perplexity."],"answer":1,"explanation":"Model and baseline scored on identical shifted targets and masks. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable."},"prerequisites":["tiny-causal-decoder","sequence-packing-loss-denominators"],"related":["tiny-causal-decoder","sequence-packing-loss-denominators","optimizer-schedule-stability","microbatch-equivalent-updates"],"sources":[{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"},{"title":"Attention Is All You Need","url":"https://arxiv.org/html/1706.03762v7"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-pretraining","subbranch":"Next-token supervision from raw sequences","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.","science":[{"title":"Next-token supervision from raw sequences · worked relation","equation":"L=-\\frac{1}{M}\\sum_{t:m_t=1}\\log p_\\theta(x_{t+1}\\mid x_{\\le t}),\\qquad \\operatorname{PPL}=e^L","explanation":"For three target probabilities [1/2,1/4,1/2], total NLL is ln2+ln4+ln2=ln16=2.772589 nats. Mean NLL is 0.924196 nats and perplexity is 16^(1/3)=2.519842. A uniform nine-token predictor has mean NLL ln9=2.197225 and perplexity 9. Feeding the answer token at the very position being scored would invalidate this comparison through target leakage.","assumptions":"Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable."}],"history":[]}