{"id":"post-training","level":"model","name":"Post-training & feedback","shortName":"Post-training & feedback","role":"Shape a pretrained model into an assistant.","description":"Pretraining builds broad predictive capability. Further supervised examples, preference learning, reinforcement learning, and other objectives can then change behavior toward useful instruction following and task performance. These methods alter model weights; a prompt alone normally does not. Feedback quality and evaluation design become central because optimizing a proxy can reward unwanted shortcuts. Different systems use different recipes, and the atlas does not assign a private recipe to Astra.\n\nChanged system behavior can come from several boundaries: updated weights, a new prompt or context, a retrieval collection, a tool contract or a runtime policy. Only the first is necessarily a model-parameter update. Evaluation should identify which artifact changed and preserve the other conditions when attributing an improvement. A runtime repair should not be described as teaching the model unless a learning procedure actually occurred.\n\nPost-training encompasses multiple contracts rather than one universal loss. Demonstration targets support supervised likelihood training; labeled response pairs support preference objectives; a learned reward or a checker supplies a signal for sampled policy updates. The same decoder can undergo several of these stages while keeping its architecture. LoRA or full tuning specifies which parameters may change, not whether the signal is SFT, DPO or reward. The executable lifecycle includes one-turn SFT, DPO and a bounded verifier bandit, with exact scope shown beside each branch.","mechanism":["Collect demonstrations or task-specific feedback.","Optimize against the chosen supervision or reward signal.","Evaluate generalization and unintended behavior on held-out tasks.","Behavioral comparisons bind model, data, prompts, tools and runtime policy to distinct versions.","A fixed task suite checks regressions after the selected change rather than attributing every outcome difference to post-training.","Identify whether the available evidence is a demonstration, pair label or scored sampled action.","State the exact loss and target/pair/rollout denominator.","Declare trainable state and the frozen reference separately from the objective."],"specs":[{"label":"Stage","value":"After base pretraining","context":"May include multiple training phases."},{"label":"Feedback","value":"Demonstrations, preferences, or task outcomes","context":"Feedback need not be objectively verifiable; the method determines how it is used."}],"interfaces":[{"name":"Base model and feedback","direction":"in","signal":"Training examples or rewards"},{"name":"Assistant model","direction":"out","signal":"Changed parameters and behavior"}],"tradeoff":"A reward is a proxy for the intended behavior. Improvements on one evaluation can coexist with regressions elsewhere.","whyAI":"Post-training connects broad modeling ability to more useful interactive behavior.","related":["loss-optimizer","evaluation","tool-loop","workflow-graphs-state","model-artifact-rollout-drift","sft-chat-loss-masks","preference-reward-model","direct-preference-objective","policy-gradient-kl-control","verifiable-reward-group-updates","reward-proxy-overoptimization","low-rank-adaptation","teacher-student-distillation"],"sources":[{"title":"Ouyang et al. — InstructGPT","url":"https://arxiv.org/abs/2203.02155"},{"title":"Building effective agents","url":"https://www.anthropic.com/engineering/building-effective-agents"},{"title":"Training language models to follow instructions with human feedback","url":"https://arxiv.org/html/2203.02155v1"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"scale":"Logical structure · mapped onto physical hardware","kind":"post-training","misconception":"Instruction following is learned behavior, not proof of perfect reliability.","check":{"question":"How does post-training differ from adding a prompt?","options":["It normally changes weights through training","It merely changes the screen color","It requires no examples or objective"],"answer":0,"explanation":"Post-training performs optimization; prompting supplies inference context."},"branch":"Learning","subbranch":"Behavioral objectives","science":[{"title":"Reward optimization with a reference penalty","equation":"max_θ E[x∼D, y∼πθ(·|x)][r(x,y)] − β·E[x∼D][D_KL(πθ(·|x) ∥ π_ref(·|x))]","explanation":"One reinforcement-learning formulation rewards useful responses while penalizing excessive movement from a reference policy. The reward r is an operational training signal; optimizing it need not maximize every property people care about. The KL term makes the tradeoff explicit rather than eliminating reward-model error.","assumptions":"This is an idealized core RLHF objective with β≥0. It omits auxiliary pretraining losses and optimizer details. Supervised fine-tuning, direct preference optimization, and learning from verifiable rewards can use different objectives; no private Astra recipe is implied."},{"title":"Choose supervision by the behavior being learned · worked relation","equation":"M_{SFT}=\\sum_i T_i^{response},\\qquad M_{DPO}=N_{pairs},\\qquad M_{RL}=N_{prompts}G","explanation":"For a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.","assumptions":"Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators."}],"history":[{"year":2022,"title":"InstructGPT combines demonstrations and preferences","significance":"Ouyang and colleagues reported supervised instruction tuning followed by preference-based reinforcement learning, showing that user preference could improve without merely increasing parameter count.","source":{"title":"Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback","url":"https://arxiv.org/abs/2203.02155"}}],"prerequisites":["loss-optimizer","evaluation"],"learningObjective":"Evaluate a reward-minus-reference-penalty objective and distinguish that score from broad model quality.","engineeringExample":"Synthetic comparison with β = 0.5: response policy A has expected reward 2 and KL divergence 0.4 nats, giving objective 1.8. Policy B has reward 1.9 and KL 0.1 nats, giving 1.85. The penalized objective favors B even though its raw reward is lower.\n\nThe same fictional malformed quantity is rejected in both incident modes. The fixed workflow stops; the authored branching policy repairs the string into an integer and continues. This behavior difference comes entirely from application control code, with no trained model or parameter update. It is useful engineering progress, but it is not evidence of post-training a language model.\n\nFor a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.","evidenceNotes":"This illustrates the idealized RLHF objective in the cited InstructGPT work, using invented expected rewards and KL values. Reward quality, the chosen reference, auxiliary objectives and deployment evaluation are separate; no private training recipe is assumed.\n\nThe incident policy is explicitly authored. The extension distinguishes behavioral system changes from actual weight adaptation and makes no claim of new post-training execution.\n\nMode: executable SFT DPO and bandit. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","checks":[{"question":"An application adds a branch that repairs malformed tool arguments, without changing model weights. What changed?","options":["The runtime policy","Necessarily the model’s post-training weights","The tokenizer’s vocabulary"],"answer":0,"explanation":"A new application branch changes system behavior independently of model training."},{"question":"Which comparison is invalid without further interpretation?","options":["Ranking SFT and DPO quality by their raw scalar loss magnitudes.","Comparing the same task accuracy at both saved checkpoints.","Inspecting the response mask in SFT."],"answer":0,"explanation":"Ranking SFT and DPO quality by their raw scalar loss magnitudes. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators."}],"labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}