Packing documents and masking valid targets

The mechanism

Packing places multiple sequences into fixed storage rows. Padding utilization, permission to attend across documents and permission to train on a target are three distinct choices. A causal mask blocks future inputs; a document mask can additionally block earlier tokens from another packed document. A target mask decides which shifted next-token predictions contribute to the objective. These masks need not be identical. Treating padding or the first token of an unrelated document as an ordinary target can change the objective even when the row has the expected shape.

Write each document with its boundary marker and shift input/target positions explicitly.
Mark causal visibility and optional same-document visibility separately.
Exclude padding and any deliberately excluded boundary targets.
Sum all included losses before dividing by the total number of included targets.

Science and assumptions

Packing documents and masking valid targets · worked relation

L=\frac{\sum_{b,t}m_{bt}\ell_{bt}}{\sum_{b,t}m_{bt}},\qquad A_{ij}=\mathbf 1[j\le i]\mathbf 1[d_j=d_i]

Draw document and target masks, then reconstruct the loss denominator of a packed batch.

Symbols and units

L: Mean included target loss
nats per target
m_bt: Target inclusion flag
dimensionless
ell_bt: Negative log probability of one target
nats
A_ij: Optional causal same-document visibility flag
dimensionless
d_i: Document identity at input position i
identifier

One sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective.

Assumptions Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Two sequences regardless of length.

Packing can reduce wasted storage while making boundary semantics harder to inspect. High utilization is not evidence of correct masking.

Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Draw document and target masks, then reconstruct the loss denominator of a packed batch.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Draw document and target masks, then reconstruct the loss denominator of a packed batch.
Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.

Interfaces

Inputs and state Input: Write each document with its boundary marker and shift input/target positions explicitly.
Result and check Output: Draw document and target masks, then reconstruct the loss denominator of a packed batch.

Check your understanding

Which denominator preserves the stated global token objective?

Two sequences regardless of length.
Eight storage slots including excluded positions.
Seven included targets.

Answer and reasoning

Seven included targets.

Seven included targets. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.

Evidence and scope

Mode: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.