{"id":"teacher-student-distillation","name":"Transfer outputs without confusing size with quality","shortName":"Transfer outputs without confusing size with quality","role":"Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.","description":"Distillation trains a student to match information supplied by a teacher. Soft probability targets retain relationships among alternatives that a single hard label discards. Temperature changes both distributions and derivative scaling; mixing soft and hard losses therefore requires a declared convention. A teacher’s generated explanation is a sequence target, while matching its class distribution is a different supervision signal. Neither is the same as quantizing stored weights. The student must still be evaluated on reserved tasks because teacher errors and transfer-set gaps can be inherited.","mechanism":["Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens.","Apply the same temperature convention to teacher and student for soft loss.","Compute the stated hard/soft objective and update only student parameters.","Compare reserved task quality, calibration and resource payload with independent denominators."],"learningObjective":"Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.","engineeringExample":"At temperature 1, teacher probabilities [0.75,0.25] and student probabilities [0.5,0.5] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: [−0.25,0.25]. Replacing the teacher distribution with hard class one changes it to [−0.5,0.5]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.","evidenceNotes":"Mode: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.","context":"Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens."},{"name":"Result and check","direction":"out","signal":"Calculate soft-target loss and gradient, then evaluate the student separately from its teacher."}],"tradeoff":"A smaller student can be easier to serve but may lose behaviors absent from transfer data. Matching the teacher does not certify independent correctness.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.","misconception":"It is a mistake to conclude: The exact teacher architecture and all its weights.","check":{"question":"What information does [0.75,0.25] retain that a hard first-class label discards?","options":["A guarantee that the first class is factually correct.","The teacher assigns nonzero relative support to the second class.","The exact teacher architecture and all its weights."],"answer":1,"explanation":"The teacher assigns nonzero relative support to the second class. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core."},"prerequisites":["output-head","evaluation"],"related":["output-head","evaluation","sft-chat-loss-masks","preference-reward-model","direct-preference-objective","policy-gradient-kl-control","verifiable-reward-group-updates","reward-proxy-overoptimization"],"sources":[{"title":"Distilling the Knowledge in a Neural Network","url":"https://arxiv.org/html/1503.02531v1"},{"title":"DeepSeek-R1","url":"https://arxiv.org/html/2501.12948v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"Transfer outputs without confusing size with quality","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.","science":[{"title":"Transfer outputs without confusing size with quality · worked relation","equation":"L_{soft}=-T^2\\sum_i p_i^{(T)}\\log q_i^{(T)},\\qquad \\frac{\\partial L_{soft}}{\\partial z_i}=T\\big(q_i^{(T)}-p_i^{(T)}\\big)","explanation":"At temperature 1, teacher probabilities [0.75,0.25] and student probabilities [0.5,0.5] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: [−0.25,0.25]. Replacing the teacher distribution with hard class one changes it to [−0.5,0.5]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.","assumptions":"Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core."}],"history":[]}