# Packing documents and masking valid targets Canonical: https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-data · Atlas v1.3.0 # Packing documents and masking valid targets Draw document and target masks, then reconstruct the loss denominator of a packed batch. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#sequence-packing-loss-denominators)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveDraw document and target masks, then reconstruct the loss denominator of a packed batch. ## The mechanism Packing places multiple sequences into fixed storage rows. Padding utilization, permission to attend across documents and permission to train on a target are three distinct choices. A causal mask blocks future inputs; a document mask can additionally block earlier tokens from another packed document. A target mask decides which shifted next-token predictions contribute to the objective. These masks need not be identical. Treating padding or the first token of an unrelated document as an ordinary target can change the objective even when the row has the expected shape. 1. Write each document with its boundary marker and shift input/target positions explicitly. 2. Mark causal visibility and optional same-document visibility separately. 3. Exclude padding and any deliberately excluded boundary targets. 4. Sum all included losses before dividing by the total number of included targets. ## Science and assumptions ### Packing documents and masking valid targets · worked relation $$ L=\frac{\sum_{b,t}m_{bt}\ell_{bt}}{\sum_{b,t}m_{bt}},\qquad A_{ij}=\mathbf 1[j\le i]\mathbf 1[d_j=d_i] $$ Draw document and target masks, then reconstruct the loss denominator of a packed batch. Symbols and units L Mean included target loss nats per target m\_bt Target inclusion flag dimensionless ell\_bt Negative log probability of one target nats A\_ij Optional causal same-document visibility flag dimensionless d\_i Document identity at input position i identifier One sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective. Assumptions Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. ## Engineering example One sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective. ## Distinctions and tradeoffs It is a mistake to conclude: Two sequences regardless of length. Packing can reduce wasted storage while making boundary semantics harder to inspect. High utilization is not evidence of correct masking. Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Draw document and target masks, then reconstruct the loss denominator of a packed batch. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Draw document and target masks, then reconstruct the loss denominator of a packed batch. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. ### Interfaces Inputs and state Input Write each document with its boundary marker and shift input/target positions explicitly. Result and check Output Draw document and target masks, then reconstruct the loss denominator of a packed batch. ## Check your understanding ### Which denominator preserves the stated global token objective? 1. Two sequences regardless of length. 2. Eight storage slots including excluded positions. 3. Seven included targets. Answer and reasoning Seven included targets. Seven included targets. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. ## Evidence and scope Mode: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. ## Sources - [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/sequence-packing-loss-denominators-5cb743a1c9061dd8.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/) - [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/) ### Related lessons - [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/) - [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/) - [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/) - [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)