# Transfer outputs without confusing size with quality Canonical: https://semiaifoundry.com/learn/atlas/teacher-student-distillation/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-posttraining · Atlas v1.3.0 # Transfer outputs without confusing size with quality Calculate soft-target loss and gradient, then evaluate the student separately from its teacher. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#teacher-student-distillation)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCalculate soft-target loss and gradient, then evaluate the student separately from its teacher. ## The mechanism Distillation trains a student to match information supplied by a teacher. Soft probability targets retain relationships among alternatives that a single hard label discards. Temperature changes both distributions and derivative scaling; mixing soft and hard losses therefore requires a declared convention. A teacher’s generated explanation is a sequence target, while matching its class distribution is a different supervision signal. Neither is the same as quantizing stored weights. The student must still be evaluated on reserved tasks because teacher errors and transfer-set gaps can be inherited. 1. Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens. 2. Apply the same temperature convention to teacher and student for soft loss. 3. Compute the stated hard/soft objective and update only student parameters. 4. Compare reserved task quality, calibration and resource payload with independent denominators. ## Science and assumptions ### Transfer outputs without confusing size with quality · worked relation $$ L_{soft}=-T^2\sum_i p_i^{(T)}\log q_i^{(T)},\qquad \frac{\partial L_{soft}}{\partial z_i}=T\big(q_i^{(T)}-p_i^{(T)}\big) $$ Calculate soft-target loss and gradient, then evaluate the student separately from its teacher. Symbols and units T Common teacher/student temperature dimensionless p\_i^(T) Teacher softened target probability dimensionless q\_i^(T) Student softened probability dimensionless z\_i Student pre-temperature logit dimensionless L\_soft Temperature-scaled soft-target loss scaled nats At temperature 1, teacher probabilities \[0.75,0.25\] and student probabilities \[0.5,0.5\] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: \[−0.25,0.25\]. Replacing the teacher distribution with hard class one changes it to \[−0.5,0.5\]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four. Assumptions Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. ## Engineering example At temperature 1, teacher probabilities \[0.75,0.25\] and student probabilities \[0.5,0.5\] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: \[−0.25,0.25\]. Replacing the teacher distribution with hard class one changes it to \[−0.5,0.5\]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four. ## Distinctions and tradeoffs It is a mistake to conclude: The exact teacher architecture and all its weights. A smaller student can be easier to serve but may lose behaviors absent from transfer data. Matching the teacher does not certify independent correctness. Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Calculate soft-target loss and gradient, then evaluate the student separately from its teacher. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Calculate soft-target loss and gradient, then evaluate the student separately from its teacher. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. ### Interfaces Inputs and state Input Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens. Result and check Output Calculate soft-target loss and gradient, then evaluate the student separately from its teacher. ## Check your understanding ### What information does \[0.75,0.25\] retain that a hard first-class label discards? 1. A guarantee that the first class is factually correct. 2. The teacher assigns nonzero relative support to the second class. 3. The exact teacher architecture and all its weights. Answer and reasoning The teacher assigns nonzero relative support to the second class. The teacher assigns nonzero relative support to the second class. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. ## Evidence and scope Mode: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. ## Sources - [Distilling the Knowledge in a Neural Network](https://arxiv.org/html/1503.02531v1) - [DeepSeek-R1](https://arxiv.org/html/2501.12948v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/teacher-student-distillation-706748f9d7909b28.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/) - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) - [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/) - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) - [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/) - [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)