# Instruction tuning and assistant-only loss

Canonical: https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

model-lifecycle-posttraining · Atlas v1.3.0

# Instruction tuning and assistant-only loss

Identify exactly which shifted response targets contribute to supervised fine-tuning.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#sft-chat-loss-masks)Mathematical, data and behavior space; logical dimensions are not physical distance

Learning objectiveIdentify exactly which shifted response targets contribute to supervised fine-tuning.

## The mechanism

Supervised fine-tuning supplies desired outputs for prompts. The model can receive both prompt and response tokens during teacher forcing while the objective trains only selected response targets. Seeing a prompt as context does not require predicting every prompt token in the loss. Role delimiters, assistant starts and end-of-turn markers require an explicit template convention; an off-by-one mask can train the wrong role or omit the terminating token. The executable lifecycle example uses one authored answer marker and supervises the answer plus EOS. Multi-turn templates are a worked extension, not an implemented chat framework.

1.  Render the exact template and retain role ownership for every token.
2.  Shift the target sequence before placing the loss mask.
3.  Include each desired assistant target and the declared end marker; exclude prompt-only targets.
4.  Divide by supervised targets and compare gradients under an intentionally wrong mask.

## Science and assumptions

### Instruction tuning and assistant-only loss · worked relation

$$
L_{SFT}=-\frac{\sum_t m_t\log p_\theta(y_t\mid x,y_{<t})}{\sum_t m_t}
$$

Identify exactly which shifted response targets contribute to supervised fine-tuning.

Symbols and units

L\_SFT

Masked supervised target loss

nats per included target

m\_t

Assistant-target inclusion flag after shifting

dimensionless

x

Prompt context

tokens

y\_t

Supervised response target

token identifier

p\_theta

Conditional target probability

dimensionless

For a six-target dialogue, prompt-target losses are \[2,3,1,2\] nats and response-target losses are \[0.4,0.6\]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.

Assumptions Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.

## Engineering example

For a six-target dialogue, prompt-target losses are \[2,3,1,2\] nats and response-target losses are \[0.4,0.6\]. All-token mean is 9/6=1.5 nats. Assistant-only mean is 1/2=0.5 nats. The prompt tokens still influence the response through attention, so their embeddings can receive gradients even though their own target losses are masked. A target mask is not a promise that all prompt-path parameters remain frozen.

## Distinctions and tradeoffs

It is a mistake to conclude: No, target masking freezes every parameter used by the prompt.

Response-only supervision focuses the direct objective while still relying on prompt representations. Template errors can silently change what is learned.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Identify exactly which shifted response targets contribute to supervised fine-tuning.

## Specifications and interfaces

Learning mode

executable one-turn response mask

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Identify exactly which shifted response targets contribute to supervised fine-tuning.

Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.

### Interfaces

Inputs and state Input

Render the exact template and retain role ownership for every token.

Result and check Output

Identify exactly which shifted response targets contribute to supervised fine-tuning.

## Check your understanding

### Can prompt embeddings receive gradients when prompt target losses are zeroed?

1.  Yes, because response predictions depend on prompt representations.
2.  No, target masking freezes every parameter used by the prompt.
3.  Only if the prompt is incorrectly used as its own target.

Answer and reasoning

Yes, because response predictions depend on prompt representations.

Yes, because response predictions depend on prompt representations. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.

## Evidence and scope

Mode: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: executable one-turn response mask. Positive included-target count and a stated role/template boundary. Core mask covers one answer and EOS; multi-turn masking here is worked-only.

## Sources

-   [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer)
-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/sft-chat-loss-masks-dd583bb8e46df0a9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Packing documents and masking valid targets](https://semiaifoundry.com/learn/atlas/sequence-packing-loss-denominators/)
-   [Next-token supervision from raw sequences](https://semiaifoundry.com/learn/atlas/causal-pretraining-objective/)
-   [Post-training & feedback](https://semiaifoundry.com/learn/atlas/post-training/)

### Related lessons

-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)
-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)
-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)
-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)
-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)