{"id":"sequence-packing-loss-denominators","name":"Packing documents and masking valid targets","shortName":"Packing documents and masking valid targets","role":"Draw document and target masks, then reconstruct the loss denominator of a packed batch.","description":"Packing places multiple sequences into fixed storage rows. Padding utilization, permission to attend across documents and permission to train on a target are three distinct choices. A causal mask blocks future inputs; a document mask can additionally block earlier tokens from another packed document. A target mask decides which shifted next-token predictions contribute to the objective. These masks need not be identical. Treating padding or the first token of an unrelated document as an ordinary target can change the objective even when the row has the expected shape.","mechanism":["Write each document with its boundary marker and shift input/target positions explicitly.","Mark causal visibility and optional same-document visibility separately.","Exclude padding and any deliberately excluded boundary targets.","Sum all included losses before dividing by the total number of included targets."],"learningObjective":"Draw document and target masks, then reconstruct the loss denominator of a packed batch.","engineeringExample":"One sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective.","evidenceNotes":"Mode: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Draw document and target masks, then reconstruct the loss denominator of a packed batch.","context":"Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Write each document with its boundary marker and shift input/target positions explicitly."},{"name":"Result and check","direction":"out","signal":"Draw document and target masks, then reconstruct the loss denominator of a packed batch."}],"tradeoff":"Packing can reduce wasted storage while making boundary semantics harder to inspect. High utilization is not evidence of correct masking.","whyAI":"Make the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Draw document and target masks, then reconstruct the loss denominator of a packed batch.","misconception":"It is a mistake to conclude: Two sequences regardless of length.","check":{"question":"Which denominator preserves the stated global token objective?","options":["Two sequences regardless of length.","Eight storage slots including excluded positions.","Seven included targets."],"answer":2,"explanation":"Seven included targets. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics."},"prerequisites":["tokenizer","distributed-gradient-weighting"],"related":["tokenizer","distributed-gradient-weighting","data-quality-dedup-audit","data-mixture-sampling","synthetic-data-selection-feedback","data-coverage-language-slices"],"sources":[{"title":"TRL SFT Trainer","url":"https://huggingface.co/docs/trl/sft_trainer"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-data","subbranch":"Packing documents and masking valid targets","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.","science":[{"title":"Packing documents and masking valid targets · worked relation","equation":"L=\\frac{\\sum_{b,t}m_{bt}\\ell_{bt}}{\\sum_{b,t}m_{bt}},\\qquad A_{ij}=\\mathbf 1[j\\le i]\\mathbf 1[d_j=d_i]","explanation":"One sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective.","assumptions":"Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics."}],"history":[]}