# Post-training & feedback Canonical: https://semiaifoundry.com/learn/atlas/post-training/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html Learning · Atlas v1.3.0 # Post-training & feedback Shape a pretrained model into an assistant. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#post-training)Logical structure · mapped onto physical hardware Learning objectiveEvaluate a reward-minus-reference-penalty objective and distinguish that score from broad model quality. ## The mechanism Pretraining builds broad predictive capability. Further supervised examples, preference learning, reinforcement learning, and other objectives can then change behavior toward useful instruction following and task performance. These methods alter model weights; a prompt alone normally does not. Feedback quality and evaluation design become central because optimizing a proxy can reward unwanted shortcuts. Different systems use different recipes, and the atlas does not assign a private recipe to Astra. Changed system behavior can come from several boundaries: updated weights, a new prompt or context, a retrieval collection, a tool contract or a runtime policy. Only the first is necessarily a model-parameter update. Evaluation should identify which artifact changed and preserve the other conditions when attributing an improvement. A runtime repair should not be described as teaching the model unless a learning procedure actually occurred. Post-training encompasses multiple contracts rather than one universal loss. Demonstration targets support supervised likelihood training; labeled response pairs support preference objectives; a learned reward or a checker supplies a signal for sampled policy updates. The same decoder can undergo several of these stages while keeping its architecture. LoRA or full tuning specifies which parameters may change, not whether the signal is SFT, DPO or reward. The executable lifecycle includes one-turn SFT, DPO and a bounded verifier bandit, with exact scope shown beside each branch. 1. Collect demonstrations or task-specific feedback. 2. Optimize against the chosen supervision or reward signal. 3. Evaluate generalization and unintended behavior on held-out tasks. 4. Behavioral comparisons bind model, data, prompts, tools and runtime policy to distinct versions. 5. A fixed task suite checks regressions after the selected change rather than attributing every outcome difference to post-training. 6. Identify whether the available evidence is a demonstration, pair label or scored sampled action. 7. State the exact loss and target/pair/rollout denominator. 8. Declare trainable state and the frozen reference separately from the objective. ## Science and assumptions ### Reward optimization with a reference penalty $$ \max_\theta\left\{\mathbb E_{\substack{x\sim D\\y\sim\pi_\theta(\cdot\mid x)}}[r(x,y)]-\beta\,\mathbb E_{x\sim D}\!\left[D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right]\right\} $$ Choose parameters to maximize expected reward minus a nonnegative coefficient times expected KL divergence from the reference policy. This preserves the source core objective, omitting the separate auxiliary pretraining term used in some InstructGPT variants. Symbols and units θ Parameters being optimized model coordinates D Prompt distribution probability distribution x Prompt sampled from D prompt y Response sampled from the policy response π\_θ Trainable conditional response policy probability distribution π\_ref Fixed reference response policy probability distribution r(x,y) Reward for prompt and response declared reward units β Nonnegative reference-penalty coefficient reward units per nat D\_KL KL divergence using natural logarithms nats One reinforcement-learning formulation rewards useful responses while penalizing excessive movement from a reference policy. The reward r is an operational training signal; optimizing it need not maximize every property people care about. The KL term makes the tradeoff explicit rather than eliminating reward-model error. Assumptions This is an idealized core RLHF objective with β≥0. It omits auxiliary pretraining losses and optimizer details. Supervised fine-tuning, direct preference optimization, and learning from verifiable rewards can use different objectives; no private Astra recipe is implied. ### Choose supervision by the behavior being learned · worked relation $$ M_{SFT}=\sum_i T_i^{response},\qquad M_{DPO}=N_{pairs},\qquad M_{RL}=N_{prompts}G $$ Choose a supervision signal and objective independently of the parameterization. Symbols and units M Objective denominator for the named stage targets pairs or rollouts T\_i^response Included response targets in demonstration i tokens N\_pairs Preference comparisons pairs N\_prompts Sampled prompts prompts G Actions sampled per prompt rollouts per prompt For a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number. Assumptions Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. ## Engineering example Synthetic comparison with β = 0.5: response policy A has expected reward 2 and KL divergence 0.4 nats, giving objective 1.8. Policy B has reward 1.9 and KL 0.1 nats, giving 1.85. The penalized objective favors B even though its raw reward is lower. The same fictional malformed quantity is rejected in both incident modes. The fixed workflow stops; the authored branching policy repairs the string into an integer and continues. This behavior difference comes entirely from application control code, with no trained model or parameter update. It is useful engineering progress, but it is not evidence of post-training a language model. For a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number. ## Distinctions and tradeoffs Instruction following is learned behavior, not proof of perfect reliability. A reward is a proxy for the intended behavior. Improvements on one evaluation can coexist with regressions elsewhere. Post-training connects broad modeling ability to more useful interactive behavior. ## Specifications and interfaces Stage After base pretraining May include multiple training phases. Feedback Demonstrations, preferences, or task outcomes Feedback need not be objectively verifiable; the method determines how it is used. ### Interfaces Base model and feedback Input Training examples or rewards Assistant model Output Changed parameters and behavior ## Check your understanding ### How does post-training differ from adding a prompt? 1. It normally changes weights through training 2. It merely changes the screen color 3. It requires no examples or objective Answer and reasoning It normally changes weights through training Post-training performs optimization; prompting supplies inference context. ### An application adds a branch that repairs malformed tool arguments, without changing model weights. What changed? 1. The runtime policy 2. Necessarily the model’s post-training weights 3. The tokenizer’s vocabulary Answer and reasoning The runtime policy A new application branch changes system behavior independently of model training. ### Which comparison is invalid without further interpretation? 1. Ranking SFT and DPO quality by their raw scalar loss magnitudes. 2. Comparing the same task accuracy at both saved checkpoints. 3. Inspecting the response mask in SFT. Answer and reasoning Ranking SFT and DPO quality by their raw scalar loss magnitudes. Ranking SFT and DPO quality by their raw scalar loss magnitudes. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. ## Historical context ### 2022 · InstructGPT combines demonstrations and preferences Ouyang and colleagues reported supervised instruction tuning followed by preference-based reinforcement learning, showing that user preference could improve without merely increasing parameter count. [Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155) ## Evidence and scope This illustrates the idealized RLHF objective in the cited InstructGPT work, using invented expected rewards and KL values. Reward quality, the chosen reference, auxiliary objectives and deployment evaluation are separate; no private training recipe is assumed. The incident policy is explicitly authored. The extension distinguishes behavioral system changes from actual weight adaptation and makes no claim of new post-training execution. Mode: executable SFT DPO and bandit. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism. ## Sources - [Ouyang et al. — InstructGPT](https://arxiv.org/abs/2203.02155) - [Building effective agents](https://www.anthropic.com/engineering/building-effective-agents) - [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1) - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/post-training-e2966da5604cf66a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Tools, memory & agent loop](https://semiaifoundry.com/learn/atlas/tool-loop/) - [Workflow graphs and explicit state](https://semiaifoundry.com/learn/atlas/workflow-graphs-state/) - [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/) - [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/) - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) - [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/) - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) - [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/) - [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/) - [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/) - [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)