{"id":"sft-chat-loss-masks","name":"Instruction tuning and assistant-only loss","shortName":"Instruction tuning and assistant-only loss","role":"Identify exactly which shifted response targets contribute to supervised fine-tuning.","description":"Supervised fine-tuning supplies desired outputs for prompts. The model can receive both prompt and response tokens during teacher forcing while the objective trains only selected response targets. Seeing a prompt as context does not require predicting every prompt token in the loss. Role delimiters, assistant starts and end-of-turn markers require an explicit template convention; an off-by-one mask can train the wrong role or omit the terminating token. The executable lifecycle example uses one authored answer marker and supervises the answer plus EOS. Multi-turn templates are a worked extension, not an implemented chat framework.","mechanism":["Render the exact template and retain role ownership for every token.","Shift the target sequence before placing the loss mask.","Include each desired assistant target and the declared end marker; exclude prompt-only targets.","Divide by supervised targets and compare gradients under an intentionally wrong mask."],"learningObjective":"Identify exactly which shifted response targets contribute to supervised fine-tuning.","engineeringExample":"For a six-target dialogue, prompt-target losses are [2,3,1,2] nats and response-target losses are [0.4,0.6]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.","evidenceNotes":"Mode: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable one-turn response mask","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Identify exactly which shifted response targets contribute to supervised fine-tuning.","context":"Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Render the exact template and retain role ownership for every token."},{"name":"Result and check","direction":"out","signal":"Identify exactly which shifted response targets contribute to supervised fine-tuning."}],"tradeoff":"Response-only supervision focuses the direct objective while still relying on prompt representations. Template errors can silently change what is learned.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Identify exactly which shifted response targets contribute to supervised fine-tuning.","misconception":"It is a mistake to conclude: No, target masking freezes every parameter used by the prompt.","check":{"question":"Can prompt embeddings receive gradients when prompt target losses are zeroed?","options":["Yes, because response predictions depend on prompt representations.","No, target masking freezes every parameter used by the prompt.","Only if the prompt is incorrectly used as its own target."],"answer":0,"explanation":"Yes, because response predictions depend on prompt representations. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only."},"prerequisites":["sequence-packing-loss-denominators","causal-pretraining-objective","post-training"],"related":["sequence-packing-loss-denominators","causal-pretraining-objective","post-training","preference-reward-model","direct-preference-objective","policy-gradient-kl-control","verifiable-reward-group-updates","reward-proxy-overoptimization"],"sources":[{"title":"TRL SFT Trainer","url":"https://huggingface.co/docs/trl/sft_trainer"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"Instruction tuning and assistant-only loss","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.","science":[{"title":"Instruction tuning and assistant-only loss · worked relation","equation":"L_{SFT}=-\\frac{\\sum_t m_t\\log p_\\theta(y_t\\mid x,y_{<t})}{\\sum_t m_t}","explanation":"For a six-target dialogue, prompt-target losses are [2,3,1,2] nats and response-target losses are [0.4,0.6]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.","assumptions":"Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only."}],"history":[]}