{
  "schemaVersion": 1,
  "id": "atlas-lesson-sequence-packing-loss-denominators",
  "title": "Packing documents and masking valid targets",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Draw document and target masks, then reconstruct the loss denominator of a packed batch.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-data",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/sequence-packing-loss-denominators-5cb743a1c9061dd8.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#sequence-packing-loss-denominators",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "be9bba309384a35b41a630de1ef2498fbb5c6d0703f931bc0322b2da628b3504",
  "contentLength": 6147,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sequence-packing-loss-denominators.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sequence-packing-loss-denominators.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Packing documents and masking valid targets",
      "start": 37,
      "end": 457
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 457,
      "end": 1376
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1376,
      "end": 1404
    },
    {
      "id": "section-4",
      "title": "Packing documents and masking valid targets · worked relation",
      "start": 1404,
      "end": 2546
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2546,
      "end": 2979
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2979,
      "end": 3414
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3414,
      "end": 3848
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3848,
      "end": 4094
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4094,
      "end": 4123
    },
    {
      "id": "section-10",
      "title": "Which denominator preserves the stated global token objective?",
      "start": 4123,
      "end": 4547
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4547,
      "end": 5160
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5160,
      "end": 5453
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5453,
      "end": 5476
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5476,
      "end": 5676
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5676,
      "end": 6147
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#sequence-packing-loss-denominators"
    },
    {
      "title": "TRL SFT Trainer",
      "url": "https://huggingface.co/docs/trl/sft_trainer"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/sequence-packing-loss-denominators-5cb743a1c9061dd8.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Tokenizer",
      "url": "https://semiaifoundry.com/learn/atlas/tokenizer/"
    },
    {
      "title": "Uneven batches and the global gradient",
      "url": "https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    },
    {
      "title": "Mixtures, epochs and exposure",
      "url": "https://semiaifoundry.com/learn/atlas/data-mixture-sampling/"
    },
    {
      "title": "Synthetic supervision and selection loops",
      "url": "https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/"
    },
    {
      "title": "Coverage across language and source slices",
      "url": "https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/"
    }
  ],
  "recordHash": "d933e16a2eafed947ab3082bd084e9d2e88afc6b2a23a14415889f2d07d4e590",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-data · Atlas v1.3.0\n\n# Packing documents and masking valid targets\n\nDraw document and target masks, then reconstruct the loss denominator of a packed batch.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#sequence-packing-loss-denominators)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveDraw document and target masks, then reconstruct the loss denominator of a packed batch.\n\n## The mechanism\n\nPacking places multiple sequences into fixed storage rows. Padding utilization, permission to attend across documents and permission to train on a target are three distinct choices. A causal mask blocks future inputs; a document mask can additionally block earlier tokens from another packed document. A target mask decides which shifted next-token predictions contribute to the objective. These masks need not be identical. Treating padding or the first token of an unrelated document as an ordinary target can change the objective even when the row has the expected shape.\n\n1.  Write each document with its boundary marker and shift input/target positions explicitly.\n2.  Mark causal visibility and optional same-document visibility separately.\n3.  Exclude padding and any deliberately excluded boundary targets.\n4.  Sum all included losses before dividing by the total number of included targets.\n\n## Science and assumptions\n\n### Packing documents and masking valid targets · worked relation\n\n$$\nL=\\frac{\\sum_{b,t}m_{bt}\\ell_{bt}}{\\sum_{b,t}m_{bt}},\\qquad A_{ij}=\\mathbf 1[j\\le i]\\mathbf 1[d_j=d_i]\n$$\n\nDraw document and target masks, then reconstruct the loss denominator of a packed batch.\n\nSymbols and units\n\nL\n\nMean included target loss\n\nnats per target\n\nm\\_bt\n\nTarget inclusion flag\n\ndimensionless\n\nell\\_bt\n\nNegative log probability of one target\n\nnats\n\nA\\_ij\n\nOptional causal same-document visibility flag\n\ndimensionless\n\nd\\_i\n\nDocument identity at input position i\n\nidentifier\n\nOne sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective.\n\nAssumptions Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.\n\n## Engineering example\n\nOne sequence contributes two valid target losses, 1 and 3 nats; another contributes five losses of 1 nat each. Token-mean loss is (1+3+5)/7=9/7=1.285714 nats. Averaging sequence means gives (2+1)/2=1.5 nats. Packing the rows into one buffer does not justify switching denominators. If a row has eight slots but only seven valid shifted targets, dividing by eight reports 1.125 and understates the objective.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Two sequences regardless of length.\n\nPacking can reduce wasted storage while making boundary semantics harder to inspect. High utilization is not evidence of correct masking.\n\nMake the effective training distribution inspectable rather than treating tokens as a homogeneous commodity. Draw document and target masks, then reconstruct the loss denominator of a packed batch.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nDraw document and target masks, then reconstruct the loss denominator of a packed batch.\n\nPositive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.\n\n### Interfaces\n\nInputs and state Input\n\nWrite each document with its boundary marker and shift input/target positions explicitly.\n\nResult and check Output\n\nDraw document and target masks, then reconstruct the loss denominator of a packed batch.\n\n## Check your understanding\n\n### Which denominator preserves the stated global token objective?\n\n1.  Two sequences regardless of length.\n2.  Eight storage slots including excluded positions.\n3.  Seven included targets.\n\nAnswer and reasoning\n\nSeven included targets.\n\nSeven included targets. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.\n\n## Evidence and scope\n\nMode: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Positive valid-target count; the displayed attention formula chooses document isolation. Other packing policies must declare their different boundary semantics.\n\n## Sources\n\n-   [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/sequence-packing-loss-denominators-5cb743a1c9061dd8.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Tokenizer](https://semiaifoundry.com/learn/atlas/tokenizer/)\n-   [Uneven batches and the global gradient](https://semiaifoundry.com/learn/atlas/distributed-gradient-weighting/)\n\n### Related lessons\n\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)\n-   [Mixtures, epochs and exposure](https://semiaifoundry.com/learn/atlas/data-mixture-sampling/)\n-   [Synthetic supervision and selection loops](https://semiaifoundry.com/learn/atlas/synthetic-data-selection-feedback/)\n-   [Coverage across language and source slices](https://semiaifoundry.com/learn/atlas/data-coverage-language-slices/)"
}