{
  "schemaVersion": 1,
  "id": "atlas-lesson-post-training",
  "title": "Post-training & feedback",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Shape a pretrained model into an assistant.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/post-training/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Learning",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This illustrates the idealized RLHF objective in the cited InstructGPT work, using invented expected rewards and KL values. Reward quality, the chosen reference, auxiliary objectives and deployment evaluation are separate; no private training recipe is assumed.\n\nThe incident policy is explicitly authored. The extension distinguishes behavioral system changes from actual weight adaptation and makes no claim of new post-training execution.\n\nMode: executable SFT DPO and bandit. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/post-training-e2966da5604cf66a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#post-training",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "384abc390b6bd76dfb11f7297a169ad4a1b55ab3ac5c2d160b2805bdf3bcd1ba",
  "contentLength": 11899,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-post-training.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-post-training.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Post-training & feedback",
      "start": 25,
      "end": 342
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 342,
      "end": 2568
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 2568,
      "end": 2596
    },
    {
      "id": "section-4",
      "title": "Reward optimization with a reference penalty",
      "start": 2596,
      "end": 4275
    },
    {
      "id": "section-5",
      "title": "Choose supervision by the behavior being learned · worked relation",
      "start": 4275,
      "end": 5391
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 5391,
      "end": 6445
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 6445,
      "end": 6757
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 6757,
      "end": 7002
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 7002,
      "end": 7136
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 7136,
      "end": 7165
    },
    {
      "id": "section-11",
      "title": "How does post-training differ from adding a prompt?",
      "start": 7165,
      "end": 7496
    },
    {
      "id": "section-12",
      "title": "An application adds a branch that repairs malformed tool arguments, without changing model weights. What changed?",
      "start": 7496,
      "end": 7845
    },
    {
      "id": "section-13",
      "title": "Which comparison is invalid without further interpretation?",
      "start": 7845,
      "end": 8415
    },
    {
      "id": "section-14",
      "title": "Historical context",
      "start": 8415,
      "end": 8438
    },
    {
      "id": "section-15",
      "title": "2022 · InstructGPT combines demonstrations and preferences",
      "start": 8438,
      "end": 8824
    },
    {
      "id": "section-16",
      "title": "Evidence and scope",
      "start": 8824,
      "end": 9861
    },
    {
      "id": "section-17",
      "title": "Sources",
      "start": 9861,
      "end": 10448
    },
    {
      "id": "section-18",
      "title": "Continue exploring",
      "start": 10448,
      "end": 10471
    },
    {
      "id": "section-19",
      "title": "Foundations",
      "start": 10471,
      "end": 10665
    },
    {
      "id": "section-20",
      "title": "Related lessons",
      "start": 10665,
      "end": 11899
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#post-training"
    },
    {
      "title": "Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback",
      "url": "https://arxiv.org/abs/2203.02155"
    },
    {
      "title": "Building effective agents",
      "url": "https://www.anthropic.com/engineering/building-effective-agents"
    },
    {
      "title": "Training language models to follow instructions with human feedback",
      "url": "https://arxiv.org/html/2203.02155v1"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/post-training-e2966da5604cf66a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Tools, memory & agent loop",
      "url": "https://semiaifoundry.com/learn/atlas/tool-loop/"
    },
    {
      "title": "Workflow graphs and explicit state",
      "url": "https://semiaifoundry.com/learn/atlas/workflow-graphs-state/"
    },
    {
      "title": "Model artifacts, rollout and drift",
      "url": "https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    },
    {
      "title": "Frozen base weights and trainable adapters",
      "url": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/"
    },
    {
      "title": "Transfer outputs without confusing size with quality",
      "url": "https://semiaifoundry.com/learn/atlas/teacher-student-distillation/"
    }
  ],
  "recordHash": "4e1bcdde4678017f80640e0660c3a2dc42dae8bf2641d7e7ed99da810a06fd7d",
  "contentFormat": "text/markdown",
  "content": "Learning · Atlas v1.3.0\n\n# Post-training & feedback\n\nShape a pretrained model into an assistant.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#post-training)Logical structure · mapped onto physical hardware\n\nLearning objectiveEvaluate a reward-minus-reference-penalty objective and distinguish that score from broad model quality.\n\n## The mechanism\n\nPretraining builds broad predictive capability. Further supervised examples, preference learning, reinforcement learning, and other objectives can then change behavior toward useful instruction following and task performance. These methods alter model weights; a prompt alone normally does not. Feedback quality and evaluation design become central because optimizing a proxy can reward unwanted shortcuts. Different systems use different recipes, and the atlas does not assign a private recipe to Astra.\n\nChanged system behavior can come from several boundaries: updated weights, a new prompt or context, a retrieval collection, a tool contract or a runtime policy. Only the first is necessarily a model-parameter update. Evaluation should identify which artifact changed and preserve the other conditions when attributing an improvement. A runtime repair should not be described as teaching the model unless a learning procedure actually occurred.\n\nPost-training encompasses multiple contracts rather than one universal loss. Demonstration targets support supervised likelihood training; labeled response pairs support preference objectives; a learned reward or a checker supplies a signal for sampled policy updates. The same decoder can undergo several of these stages while keeping its architecture. LoRA or full tuning specifies which parameters may change, not whether the signal is SFT, DPO or reward. The executable lifecycle includes one-turn SFT, DPO and a bounded verifier bandit, with exact scope shown beside each branch.\n\n1.  Collect demonstrations or task-specific feedback.\n2.  Optimize against the chosen supervision or reward signal.\n3.  Evaluate generalization and unintended behavior on held-out tasks.\n4.  Behavioral comparisons bind model, data, prompts, tools and runtime policy to distinct versions.\n5.  A fixed task suite checks regressions after the selected change rather than attributing every outcome difference to post-training.\n6.  Identify whether the available evidence is a demonstration, pair label or scored sampled action.\n7.  State the exact loss and target/pair/rollout denominator.\n8.  Declare trainable state and the frozen reference separately from the objective.\n\n## Science and assumptions\n\n### Reward optimization with a reference penalty\n\n$$\n\\max_\\theta\\left\\{\\mathbb E_{\\substack{x\\sim D\\\\y\\sim\\pi_\\theta(\\cdot\\mid x)}}[r(x,y)]-\\beta\\,\\mathbb E_{x\\sim D}\\!\\left[D_{\\mathrm{KL}}\\!\\left(\\pi_\\theta(\\cdot\\mid x)\\,\\Vert\\,\\pi_{\\mathrm{ref}}(\\cdot\\mid x)\\right)\\right]\\right\\}\n$$\n\nChoose parameters to maximize expected reward minus a nonnegative coefficient times expected KL divergence from the reference policy.\n\nThis preserves the source core objective, omitting the separate auxiliary pretraining term used in some InstructGPT variants.\n\nSymbols and units\n\nθ\n\nParameters being optimized\n\nmodel coordinates\n\nD\n\nPrompt distribution\n\nprobability distribution\n\nx\n\nPrompt sampled from D\n\nprompt\n\ny\n\nResponse sampled from the policy\n\nresponse\n\nπ\\_θ\n\nTrainable conditional response policy\n\nprobability distribution\n\nπ\\_ref\n\nFixed reference response policy\n\nprobability distribution\n\nr(x,y)\n\nReward for prompt and response\n\ndeclared reward units\n\nβ\n\nNonnegative reference-penalty coefficient\n\nreward units per nat\n\nD\\_KL\n\nKL divergence using natural logarithms\n\nnats\n\nOne reinforcement-learning formulation rewards useful responses while penalizing excessive movement from a reference policy. The reward r is an operational training signal; optimizing it need not maximize every property people care about. The KL term makes the tradeoff explicit rather than eliminating reward-model error.\n\nAssumptions This is an idealized core RLHF objective with β≥0. It omits auxiliary pretraining losses and optimizer details. Supervised fine-tuning, direct preference optimization, and learning from verifiable rewards can use different objectives; no private Astra recipe is implied.\n\n### Choose supervision by the behavior being learned · worked relation\n\n$$\nM_{SFT}=\\sum_i T_i^{response},\\qquad M_{DPO}=N_{pairs},\\qquad M_{RL}=N_{prompts}G\n$$\n\nChoose a supervision signal and objective independently of the parameterization.\n\nSymbols and units\n\nM\n\nObjective denominator for the named stage\n\ntargets pairs or rollouts\n\nT\\_i^response\n\nIncluded response targets in demonstration i\n\ntokens\n\nN\\_pairs\n\nPreference comparisons\n\npairs\n\nN\\_prompts\n\nSampled prompts\n\nprompts\n\nG\n\nActions sampled per prompt\n\nrollouts per prompt\n\nFor a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.\n\nAssumptions Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators.\n\n## Engineering example\n\nSynthetic comparison with β = 0.5: response policy A has expected reward 2 and KL divergence 0.4 nats, giving objective 1.8. Policy B has reward 1.9 and KL 0.1 nats, giving 1.85. The penalized objective favors B even though its raw reward is lower.\n\nThe same fictional malformed quantity is rejected in both incident modes. The fixed workflow stops; the authored branching policy repairs the string into an integer and continues. This behavior difference comes entirely from application control code, with no trained model or parameter update. It is useful engineering progress, but it is not evidence of post-training a language model.\n\nFor a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.\n\n## Distinctions and tradeoffs\n\nInstruction following is learned behavior, not proof of perfect reliability.\n\nA reward is a proxy for the intended behavior. Improvements on one evaluation can coexist with regressions elsewhere.\n\nPost-training connects broad modeling ability to more useful interactive behavior.\n\n## Specifications and interfaces\n\nStage\n\nAfter base pretraining\n\nMay include multiple training phases.\n\nFeedback\n\nDemonstrations, preferences, or task outcomes\n\nFeedback need not be objectively verifiable; the method determines how it is used.\n\n### Interfaces\n\nBase model and feedback Input\n\nTraining examples or rewards\n\nAssistant model Output\n\nChanged parameters and behavior\n\n## Check your understanding\n\n### How does post-training differ from adding a prompt?\n\n1.  It normally changes weights through training\n2.  It merely changes the screen color\n3.  It requires no examples or objective\n\nAnswer and reasoning\n\nIt normally changes weights through training\n\nPost-training performs optimization; prompting supplies inference context.\n\n### An application adds a branch that repairs malformed tool arguments, without changing model weights. What changed?\n\n1.  The runtime policy\n2.  Necessarily the model’s post-training weights\n3.  The tokenizer’s vocabulary\n\nAnswer and reasoning\n\nThe runtime policy\n\nA new application branch changes system behavior independently of model training.\n\n### Which comparison is invalid without further interpretation?\n\n1.  Ranking SFT and DPO quality by their raw scalar loss magnitudes.\n2.  Comparing the same task accuracy at both saved checkpoints.\n3.  Inspecting the response mask in SFT.\n\nAnswer and reasoning\n\nRanking SFT and DPO quality by their raw scalar loss magnitudes.\n\nRanking SFT and DPO quality by their raw scalar loss magnitudes. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators.\n\n## Historical context\n\n### 2022 · InstructGPT combines demonstrations and preferences\n\nOuyang and colleagues reported supervised instruction tuning followed by preference-based reinforcement learning, showing that user preference could improve without merely increasing parameter count.\n\n[Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)\n\n## Evidence and scope\n\nThis illustrates the idealized RLHF objective in the cited InstructGPT work, using invented expected rewards and KL values. Reward quality, the chosen reference, auxiliary objectives and deployment evaluation are separate; no private training recipe is assumed.\n\nThe incident policy is explicitly authored. The extension distinguishes behavioral system changes from actual weight adaptation and makes no claim of new post-training execution.\n\nMode: executable SFT DPO and bandit. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [Ouyang et al. — InstructGPT](https://arxiv.org/abs/2203.02155)\n-   [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents)\n-   [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/post-training-e2966da5604cf66a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Tools, memory & agent loop](https://semiaifoundry.com/learn/atlas/tool-loop/)\n-   [Workflow graphs and explicit state](https://semiaifoundry.com/learn/atlas/workflow-graphs-state/)\n-   [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/)\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)\n-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)\n-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)"
}