{"id":"policy-gradient-kl-control","name":"Rollouts, advantages and PPO constraints","shortName":"Rollouts, advantages and PPO constraints","role":"Separate sampled advantages, old-policy ratios, clipping and reference KL.","description":"A policy-gradient update learns from actions sampled under a recorded policy. The score-function derivative weights log-probability gradients by an advantage; the sampled action and its assigned reward are treated as data for that update. PPO adds a clipped probability-ratio surrogate for reusing rollout data near the old policy. The old rollout policy and a longer-lived reference policy play different roles. Clipping the surrogate is not a hard guarantee that every new probability ratio or KL stays bounded. The lifecycle core instead implements a two-action REINFORCE update with an exact detached baseline and categorical KL; PPO is worked here only.","mechanism":["Retain the identity and action probabilities of the policy that generated each rollout.","Compute a stated return/baseline advantage and detach it for the score-function term.","For PPO, compare current/old ratios with the clipping interval and respect the sign of the advantage.","Compute reference KL under its explicit distribution and check actual task return independently."],"learningObjective":"Separate sampled advantages, old-policy ratios, clipping and reference KL.","engineeringExample":"For a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL([0.7,0.3] || [0.5,0.5]) is 0.082283 nats, not 0.2 and not the ratio 1.4.","evidenceNotes":"Mode: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Separate sampled advantages, old-policy ratios, clipping and reference KL.","context":"One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Retain the identity and action probabilities of the policy that generated each rollout."},{"name":"Result and check","direction":"out","signal":"Separate sampled advantages, old-policy ratios, clipping and reference KL."}],"tradeoff":"Reusing rollouts can reduce sampling demand while increasing sensitivity to stale policies and surrogate choices. A clipped objective does not replace measured policy drift.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Separate sampled advantages, old-policy ratios, clipping and reference KL.","misconception":"It is a mistake to conclude: −2.4, because all ratios are always forcibly clamped.","check":{"question":"For advantage −2 and ratio 1.4, what is the clipped-surrogate contribution?","options":["−2.8, because the minimum retains the worse contribution.","−2.4, because all ratios are always forcibly clamped.","+2.4, because reward signs are ignored."],"answer":0,"explanation":"−2.8, because the minimum retains the worse contribution. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO."},"prerequisites":["preference-reward-model","output-head"],"related":["preference-reward-model","output-head","sft-chat-loss-masks","direct-preference-objective","verifiable-reward-group-updates","reward-proxy-overoptimization","low-rank-adaptation","teacher-student-distillation"],"sources":[{"title":"Proximal Policy Optimization Algorithms","url":"https://arxiv.org/pdf/1707.06347"},{"title":"Training language models to follow instructions with human feedback","url":"https://arxiv.org/html/2203.02155v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"Rollouts, advantages and PPO constraints","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.","science":[{"title":"Rollouts, advantages and PPO constraints · worked relation","equation":"J_{clip}=\\min(rA,\\operatorname{clip}(r,1-\\epsilon,1+\\epsilon)A),\\quad r=\\frac{\\pi_\\theta(a\\mid s)}{\\pi_{old}(a\\mid s)},\\quad D_{KL}(p\\Vert q)=\\sum_i p_i\\log\\frac{p_i}{q_i}","explanation":"For a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL([0.7,0.3] || [0.5,0.5]) is 0.082283 nats, not 0.2 and not the ratio 1.4.","assumptions":"One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO."}],"history":[]}