Next-token supervision from raw sequences

The mechanism

Autoregressive pretraining asks a model to predict each next token from earlier inputs. The target is the shifted sequence, not the token already visible at the same position. Summing negative log probabilities gives a sequence score; averaging included targets gives the training objective. Exponentiating this mean yields perplexity under the same tokenization and mask. Answer accuracy is a different measurement: it observes selected decisions rather than every token probability. A model can become better at punctuation or repeated format markers while its answer decisions remain unchanged.

Pair input position t with token t+1 and mark valid targets.
Compute stable log-softmax and gather each target’s log probability.
Sum losses and divide by included targets across the batch.
Fit a simple baseline on training data only and compare the same reserved targets.

Science and assumptions

Next-token supervision from raw sequences · worked relation

L=-\frac{1}{M}\sum_{t:m_t=1}\log p_\theta(x_{t+1}\mid x_{\le t}),\qquad \operatorname{PPL}=e^L

Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.

Symbols and units

L: Mean included next-token NLL
nats per token
M: Included target count
tokens
m_t: Target inclusion flag
dimensionless
p_theta: Conditional target probability
dimensionless
PPL: Perplexity under this tokenization
dimensionless

For three target probabilities [1/2,1/4,1/2], total NLL is ln2+ln4+ln2=ln16=2.772589 nats. Mean NLL is 0.924196 nats and perplexity is 16^(1/3)=2.519842. A uniform nine-token predictor has mean NLL ln9=2.197225 and perplexity 9. Feeding the answer token at the very position being scored would invalidate this comparison through target leakage.

Assumptions Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Model answer accuracy versus baseline perplexity.

Token likelihood supplies dense supervision but weights frequent formatting and rare task-critical targets according to their actual counts.

Turn an objective into a stable, restartable experiment with controlled budgets. Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.

Specifications and interfaces

Learning mode: executable core
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.
Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.

Interfaces

Inputs and state Input: Pair input position t with token t+1 and mark valid targets.
Result and check Output: Construct shifted next-token targets and interpret token likelihood separately from answer accuracy.

Check your understanding

Which comparison uses the same next-token objective?

Training loss after showing the target at its own input position.
Model and baseline scored on identical shifted targets and masks.
Model answer accuracy versus baseline perplexity.

Answer and reasoning

Model and baseline scored on identical shifted targets and masks.

Model and baseline scored on identical shifted targets and masks. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.

Evidence and scope

Mode: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: executable core. Natural logarithms, positive target probabilities and nonzero denominator. Perplexities with different tokenization/masks are not directly comparable.