{
  "schemaVersion": 1,
  "id": "atlas-lesson-teacher-student-distillation",
  "title": "Transfer outputs without confusing size with quality",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/teacher-student-distillation/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/teacher-student-distillation-706748f9d7909b28.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#teacher-student-distillation",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "4ab84fa3789668cc018982de88716a124dfdd07c9cc8deadb0477e5b7e162d8e",
  "contentLength": 6875,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-teacher-student-distillation.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-teacher-student-distillation.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Transfer outputs without confusing size with quality",
      "start": 45,
      "end": 482
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 482,
      "end": 1475
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1475,
      "end": 1503
    },
    {
      "id": "section-4",
      "title": "Transfer outputs without confusing size with quality · worked relation",
      "start": 1503,
      "end": 2719
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2719,
      "end": 3179
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3179,
      "end": 3641
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3641,
      "end": 4086
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4086,
      "end": 4357
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4357,
      "end": 4386
    },
    {
      "id": "section-10",
      "title": "What information does \\[0.75,0.25\\] retain that a hard first-class label discards?",
      "start": 4386,
      "end": 4981
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4981,
      "end": 5602
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5602,
      "end": 5965
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5965,
      "end": 5988
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5988,
      "end": 6174
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6174,
      "end": 6875
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#teacher-student-distillation"
    },
    {
      "title": "Distilling the Knowledge in a Neural Network",
      "url": "https://arxiv.org/html/1503.02531v1"
    },
    {
      "title": "DeepSeek-R1",
      "url": "https://arxiv.org/html/2501.12948v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/teacher-student-distillation-706748f9d7909b28.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    }
  ],
  "recordHash": "bb8354cc1758d866a1fcce6565e7b715e1db8435893d9eebbd704ec9041b8e80",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Transfer outputs without confusing size with quality\n\nCalculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#teacher-student-distillation)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCalculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\n## The mechanism\n\nDistillation trains a student to match information supplied by a teacher. Soft probability targets retain relationships among alternatives that a single hard label discards. Temperature changes both distributions and derivative scaling; mixing soft and hard losses therefore requires a declared convention. A teacher’s generated explanation is a sequence target, while matching its class distribution is a different supervision signal. Neither is the same as quantizing stored weights. The student must still be evaluated on reserved tasks because teacher errors and transfer-set gaps can be inherited.\n\n1.  Record teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens.\n2.  Apply the same temperature convention to teacher and student for soft loss.\n3.  Compute the stated hard/soft objective and update only student parameters.\n4.  Compare reserved task quality, calibration and resource payload with independent denominators.\n\n## Science and assumptions\n\n### Transfer outputs without confusing size with quality · worked relation\n\n$$\nL_{soft}=-T^2\\sum_i p_i^{(T)}\\log q_i^{(T)},\\qquad \\frac{\\partial L_{soft}}{\\partial z_i}=T\\big(q_i^{(T)}-p_i^{(T)}\\big)\n$$\n\nCalculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\nSymbols and units\n\nT\n\nCommon teacher/student temperature\n\ndimensionless\n\np\\_i^(T)\n\nTeacher softened target probability\n\ndimensionless\n\nq\\_i^(T)\n\nStudent softened probability\n\ndimensionless\n\nz\\_i\n\nStudent pre-temperature logit\n\ndimensionless\n\nL\\_soft\n\nTemperature-scaled soft-target loss\n\nscaled nats\n\nAt temperature 1, teacher probabilities \\[0.75,0.25\\] and student probabilities \\[0.5,0.5\\] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: \\[−0.25,0.25\\]. Replacing the teacher distribution with hard class one changes it to \\[−0.5,0.5\\]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.\n\nAssumptions Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.\n\n## Engineering example\n\nAt temperature 1, teacher probabilities \\[0.75,0.25\\] and student probabilities \\[0.5,0.5\\] give cross-entropy ln2=0.693147 nats. The student-logit gradient is student minus teacher: \\[−0.25,0.25\\]. Replacing the teacher distribution with hard class one changes it to \\[−0.5,0.5\\]. At temperature 2 with the same softened probabilities, an unscaled soft loss has half that logit gradient; a T²-scaled convention multiplies it by four.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: The exact teacher architecture and all its weights.\n\nA smaller student can be easier to serve but may lose behaviors absent from transfer data. Matching the teacher does not certify independent correctness.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Calculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCalculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\nTeacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.\n\n### Interfaces\n\nInputs and state Input\n\nRecord teacher identity, transfer inputs and whether targets are logits, probabilities or generated tokens.\n\nResult and check Output\n\nCalculate soft-target loss and gradient, then evaluate the student separately from its teacher.\n\n## Check your understanding\n\n### What information does \\[0.75,0.25\\] retain that a hard first-class label discards?\n\n1.  A guarantee that the first class is factually correct.\n2.  The teacher assigns nonzero relative support to the second class.\n3.  The exact teacher architecture and all its weights.\n\nAnswer and reasoning\n\nThe teacher assigns nonzero relative support to the second class.\n\nThe teacher assigns nonzero relative support to the second class. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.\n\n## Evidence and scope\n\nMode: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Teacher probabilities detached; fixed shared temperature. T² scaling is explicit. Worked calculation only; no student distillation trainer is in the lifecycle core.\n\n## Sources\n\n-   [Distilling the Knowledge in a Neural Network](https://arxiv.org/html/1503.02531v1)\n-   [DeepSeek-R1](https://arxiv.org/html/2501.12948v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/teacher-student-distillation-706748f9d7909b28.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)"
}