The mechanism
Distillation trains a student to match information supplied by a teacher. Soft probability targets retain relationships among alternatives that a single hard label discards. Temperature changes both distributions and derivative scaling; mixing soft and hard losses therefore requires a declared convention. A teacher’s generated explanation is a sequence target, while matching its class distribution is a different supervision signal. Neither is the same as quantizing stored weights. The student must still be evaluated on reserved tasks because teacher errors and transfer-set gaps can be inherited.
- Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens.
- Apply the same temperature convention to teacher and student for soft loss.
- Compute the stated hard/soft objective and update only student parameters.
- Compare reserved task quality, calibration and resource payload with independent denominators.
Science and assumptions
Transfer outputs without confusing size with quality · worked relation
Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.
Symbols and units
- T
- Common teacher/student temperature
dimensionless
- p_i^(T)
- Teacher softened target probability
dimensionless
- q_i^(T)
- Student softened probability
dimensionless
- z_i
- Student pre-temperature logit
dimensionless
- L_soft
- Temperature-scaled soft-target loss
scaled nats
At temperature 1, teacher probabilities [0.75,0.25] and student probabilities [0.5,0.5] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: [−0.25,0.25]. Replacing the teacher distribution with hard class one changes it to [−0.5,0.5]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.
Assumptions Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.
Engineering example
At temperature 1, teacher probabilities [0.75,0.25] and student probabilities [0.5,0.5] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: [−0.25,0.25]. Replacing the teacher distribution with hard class one changes it to [−0.5,0.5]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.
Distinctions and tradeoffs
It is a mistake to conclude: The exact teacher architecture and all its weights.
A smaller student can be easier to serve but may lose behaviors absent from transfer data. Matching the teacher does not certify independent correctness.
Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.
Specifications and interfaces
- Learning mode
- worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.
Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.
Interfaces
- Inputs and state Input
- Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens.
- Result and check Output
- Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.
Check your understanding
What information does [0.75,0.25] retain that a hard first-class label discards?
- A guarantee that the first class is factually correct.
- The teacher assigns nonzero relative support to the second class.
- The exact teacher architecture and all its weights.
Answer and reasoning
The teacher assigns nonzero relative support to the second class.
The teacher assigns nonzero relative support to the second class. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.
Evidence and scope
Mode: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.