{
  "schemaVersion": 1,
  "id": "atlas-lesson-sft-chat-loss-masks",
  "title": "Instruction tuning and assistant-only loss",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Identify exactly which shifted response targets contribute to supervised fine-tuning.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/sft-chat-loss-masks-dd583bb8e46df0a9.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#sft-chat-loss-masks",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "3a7b78129721bb2710d177ace97fc5b9ea1fa722824a6d6ef962fd1faba53d1a",
  "contentLength": 6690,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sft-chat-loss-masks.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sft-chat-loss-masks.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Instruction tuning and assistant-only loss",
      "start": 45,
      "end": 443
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 443,
      "end": 1425
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1425,
      "end": 1453
    },
    {
      "id": "section-4",
      "title": "Instruction tuning and assistant-only loss · worked relation",
      "start": 1453,
      "end": 2547
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2547,
      "end": 2988
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2988,
      "end": 3452
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3452,
      "end": 3886
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3886,
      "end": 4108
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4108,
      "end": 4137
    },
    {
      "id": "section-10",
      "title": "Can prompt embeddings receive gradients when prompt target losses are zeroed?",
      "start": 4137,
      "end": 4727
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4727,
      "end": 5346
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5346,
      "end": 5730
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5730,
      "end": 5753
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5753,
      "end": 6098
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6098,
      "end": 6690
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#sft-chat-loss-masks"
    },
    {
      "title": "TRL SFT Trainer",
      "url": "https://huggingface.co/docs/trl/sft_trainer"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/sft-chat-loss-masks-dd583bb8e46df0a9.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Packing documents and masking valid targets",
      "url": "https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/"
    },
    {
      "title": "Next-token supervision from raw sequences",
      "url": "https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/"
    },
    {
      "title": "Post-training & feedback",
      "url": "https://semiaifoundry.com/learn/atlas/post-training/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    }
  ],
  "recordHash": "4179373c4f25a451ea2ba0318b9a45658ac1ce9bf5d8c83e358550bf67a1cfcd",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Instruction tuning and assistant-only loss\n\nIdentify exactly which shifted response targets contribute to supervised fine-tuning.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#sft-chat-loss-masks)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveIdentify exactly which shifted response targets contribute to supervised fine-tuning.\n\n## The mechanism\n\nSupervised fine-tuning supplies desired outputs for prompts. The model can receive both prompt and response tokens during teacher forcing while the objective trains only selected response targets. Seeing a prompt as context does not require predicting every prompt token in the loss. Role delimiters, assistant starts and end-of-turn markers require an explicit template convention; an off-by-one mask can train the wrong role or omit the terminating token. The executable lifecycle example uses one authored answer marker and supervises the answer plus EOS. Multi-turn templates are a worked extension, not an implemented chat framework.\n\n1.  Render the exact template and retain role ownership for every token.\n2.  Shift the target sequence before placing the loss mask.\n3.  Include each desired assistant target and the declared end marker; exclude prompt-only targets.\n4.  Divide by supervised targets and compare gradients under an intentionally wrong mask.\n\n## Science and assumptions\n\n### Instruction tuning and assistant-only loss · worked relation\n\n$$\nL_{SFT}=-\\frac{\\sum_t m_t\\log p_\\theta(y_t\\mid x,y_{<t})}{\\sum_t m_t}\n$$\n\nIdentify exactly which shifted response targets contribute to supervised fine-tuning.\n\nSymbols and units\n\nL\\_SFT\n\nMasked supervised target loss\n\nnats per included target\n\nm\\_t\n\nAssistant-target inclusion flag after shifting\n\ndimensionless\n\nx\n\nPrompt context\n\ntokens\n\ny\\_t\n\nSupervised response target\n\ntoken identifier\n\np\\_theta\n\nConditional target probability\n\ndimensionless\n\nFor a six-target dialogue, prompt-target losses are \\[2,3,1,2\\] nats and response-target losses are \\[0.4,0.6\\]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.\n\nAssumptions Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.\n\n## Engineering example\n\nFor a six-target dialogue, prompt-target losses are \\[2,3,1,2\\] nats and response-target losses are \\[0.4,0.6\\]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: No, target masking freezes every parameter used by the prompt.\n\nResponse-only supervision focuses the direct objective while still relying on prompt representations. Template errors can silently change what is learned.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Identify exactly which shifted response targets contribute to supervised fine-tuning.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable one-turn response mask\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nIdentify exactly which shifted response targets contribute to supervised fine-tuning.\n\nPositive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.\n\n### Interfaces\n\nInputs and state Input\n\nRender the exact template and retain role ownership for every token.\n\nResult and check Output\n\nIdentify exactly which shifted response targets contribute to supervised fine-tuning.\n\n## Check your understanding\n\n### Can prompt embeddings receive gradients when prompt target losses are zeroed?\n\n1.  Yes, because response predictions depend on prompt representations.\n2.  No, target masking freezes every parameter used by the prompt.\n3.  Only if the prompt is incorrectly used as its own target.\n\nAnswer and reasoning\n\nYes, because response predictions depend on prompt representations.\n\nYes, because response predictions depend on prompt representations. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.\n\n## Evidence and scope\n\nMode: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.\n\n## Sources\n\n-   [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/sft-chat-loss-masks-dd583bb8e46df0a9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)\n-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)\n-   [Post-training & feedback](https://semiaifoundry.com/learn/atlas/post-training/)\n\n### Related lessons\n\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)"
}