semiAIfoundry

Learning · Atlas v1.3.0

Post-training & feedback

Shape a pretrained model into an assistant.

Explore in AtlasLogical structure · mapped onto physical hardware

Learning objectiveEvaluate a reward-minus-reference-penalty objective and distinguish that score from broad model quality.

The mechanism

Pretraining builds broad predictive capability. Further supervised examples, preference learning, reinforcement learning, and other objectives can then change behavior toward useful instruction following and task performance. These methods alter model weights; a prompt alone normally does not. Feedback quality and evaluation design become central because optimizing a proxy can reward unwanted shortcuts. Different systems use different recipes, and the atlas does not assign a private recipe to Astra.

Changed system behavior can come from several boundaries: updated weights, a new prompt or context, a retrieval collection, a tool contract or a runtime policy. Only the first is necessarily a model-parameter update. Evaluation should identify which artifact changed and preserve the other conditions when attributing an improvement. A runtime repair should not be described as teaching the model unless a learning procedure actually occurred.

Post-training encompasses multiple contracts rather than one universal loss. Demonstration targets support supervised likelihood training; labeled response pairs support preference objectives; a learned reward or a checker supplies a signal for sampled policy updates. The same decoder can undergo several of these stages while keeping its architecture. LoRA or full tuning specifies which parameters may change, not whether the signal is SFT, DPO or reward. The executable lifecycle includes one-turn SFT, DPO and a bounded verifier bandit, with exact scope shown beside each branch.

  1. Collect demonstrations or task-specific feedback.
  2. Optimize against the chosen supervision or reward signal.
  3. Evaluate generalization and unintended behavior on held-out tasks.
  4. Behavioral comparisons bind model, data, prompts, tools and runtime policy to distinct versions.
  5. A fixed task suite checks regressions after the selected change rather than attributing every outcome difference to post-training.
  6. Identify whether the available evidence is a demonstration, pair label or scored sampled action.
  7. State the exact loss and target/pair/rollout denominator.
  8. Declare trainable state and the frozen reference separately from the objective.

Science and assumptions

Reward optimization with a reference penalty

max⁡θ{Ex∼Dy∼πθ(⋅∣x)[r(x,y)]−β Ex∼D ⁣[DKL ⁣(πθ(⋅∣x) ∥ πref(⋅∣x))]}\max_\theta\left\{\mathbb E_{\substack{x\sim D\\y\sim\pi_\theta(\cdot\mid x)}}[r(x,y)]-\beta\,\mathbb E_{x\sim D}\!\left[D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right]\right\}

Choose parameters to maximize expected reward minus a nonnegative coefficient times expected KL divergence from the reference policy.

This preserves the source core objective, omitting the separate auxiliary pretraining term used in some InstructGPT variants.

Symbols and units
θ
Parameters being optimized

model coordinates

D
Prompt distribution

probability distribution

x
Prompt sampled from D

prompt

y
Response sampled from the policy

response

π_θ
Trainable conditional response policy

probability distribution

π_ref
Fixed reference response policy

probability distribution

r(x,y)
Reward for prompt and response

declared reward units

β
Nonnegative reference-penalty coefficient

reward units per nat

D_KL
KL divergence using natural logarithms

nats

One reinforcement-learning formulation rewards useful responses while penalizing excessive movement from a reference policy. The reward r is an operational training signal; optimizing it need not maximize every property people care about. The KL term makes the tradeoff explicit rather than eliminating reward-model error.

Assumptions This is an idealized core RLHF objective with β≥0. It omits auxiliary pretraining losses and optimizer details. Supervised fine-tuning, direct preference optimization, and learning from verifiable rewards can use different objectives; no private Astra recipe is implied.

Choose supervision by the behavior being learned · worked relation

MSFT=∑iTiresponse,MDPO=Npairs,MRL=NpromptsGM_{SFT}=\sum_i T_i^{response},\qquad M_{DPO}=N_{pairs},\qquad M_{RL}=N_{prompts}G

Choose a supervision signal and objective independently of the parameterization.

Symbols and units
M
Objective denominator for the named stage

targets pairs or rollouts

T_i^response
Included response targets in demonstration i

tokens

N_pairs
Preference comparisons

pairs

N_prompts
Sampled prompts

prompts

G
Actions sampled per prompt

rollouts per prompt

For a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.

Assumptions Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators.

Engineering example

Synthetic comparison with β = 0.5: response policy A has expected reward 2 and KL divergence 0.4 nats, giving objective 1.8. Policy B has reward 1.9 and KL 0.1 nats, giving 1.85. The penalized objective favors B even though its raw reward is lower.

The same fictional malformed quantity is rejected in both incident modes. The fixed workflow stops; the authored branching policy repairs the string into an integer and continues. This behavior difference comes entirely from application control code, with no trained model or parameter update. It is useful engineering progress, but it is not evidence of post-training a language model.

For a batch of two one-turn demonstrations with answer plus EOS, SFT averages four target losses. Two DPO comparisons instead average two pair losses. The core reward branch samples four answers for each of two prompts and averages eight rollout terms. Their numerical loss values have different denominators and meanings; a lower DPO number is not directly better than a higher SFT number.

Distinctions and tradeoffs

Instruction following is learned behavior, not proof of perfect reliability.

A reward is a proxy for the intended behavior. Improvements on one evaluation can coexist with regressions elsewhere.

Post-training connects broad modeling ability to more useful interactive behavior.

Specifications and interfaces

Stage
After base pretraining

May include multiple training phases.

Feedback
Demonstrations, preferences, or task outcomes

Feedback need not be objectively verifiable; the method determines how it is used.

Interfaces

Base model and feedback Input
Training examples or rewards
Assistant model Output
Changed parameters and behavior

Check your understanding

How does post-training differ from adding a prompt?

  1. It normally changes weights through training
  2. It merely changes the screen color
  3. It requires no examples or objective
Answer and reasoning

It normally changes weights through training

Post-training performs optimization; prompting supplies inference context.

An application adds a branch that repairs malformed tool arguments, without changing model weights. What changed?

  1. The runtime policy
  2. Necessarily the model’s post-training weights
  3. The tokenizer’s vocabulary
Answer and reasoning

The runtime policy

A new application branch changes system behavior independently of model training.

Which comparison is invalid without further interpretation?

  1. Ranking SFT and DPO quality by their raw scalar loss magnitudes.
  2. Comparing the same task accuracy at both saved checkpoints.
  3. Inspecting the response mask in SFT.
Answer and reasoning

Ranking SFT and DPO quality by their raw scalar loss magnitudes.

Ranking SFT and DPO quality by their raw scalar loss magnitudes. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators.

Historical context

2022 · InstructGPT combines demonstrations and preferences

Ouyang and colleagues reported supervised instruction tuning followed by preference-based reinforcement learning, showing that user preference could improve without merely increasing parameter count.

Ouyang et al. — Training Language Models to Follow Instructions with Human Feedback

Evidence and scope

This illustrates the idealized RLHF objective in the cited InstructGPT work, using invented expected rewards and KL values. Reward quality, the chosen reference, auxiliary objectives and deployment evaluation are separate; no private training recipe is assumed.

The incident policy is explicitly authored. The extension distinguishes behavioral system changes from actual weight adaptation and makes no claim of new post-training execution.

Mode: executable SFT DPO and bandit. Core conventions: answer plus EOS SFT, response-pair DPO, one-action bandit rollouts. These are not a claim that every post-training implementation uses the same denominators. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.

Sources

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

Related lessons