Rollouts, advantages and PPO constraints

The mechanism

A policy-gradient update learns from actions sampled under a recorded policy. The score-function derivative weights log-probability gradients by an advantage; the sampled action and its assigned reward are treated as data for that update. PPO adds a clipped probability-ratio surrogate for reusing rollout data near the old policy. The old rollout policy and a longer-lived reference policy play different roles. Clipping the surrogate is not a hard guarantee that every new probability ratio or KL stays bounded. The lifecycle core instead implements a two-action REINFORCE update with an exact detached baseline and categorical KL; PPO is worked here only.

Retain the identity and action probabilities of the policy that generated each rollout.
Compute a stated return/baseline advantage and detach it for the score-function term.
For PPO, compare current/old ratios with the clipping interval and respect the sign of the advantage.
Compute reference KL under its explicit distribution and check actual task return independently.

Science and assumptions

Rollouts, advantages and PPO constraints · worked relation

J_{clip}=\min(rA,\operatorname{clip}(r,1-\epsilon,1+\epsilon)A),\quad r=\frac{\pi_\theta(a\mid s)}{\pi_{old}(a\mid s)},\quad D_{KL}(p\Vert q)=\sum_i p_i\log\frac{p_i}{q_i}

Separate sampled advantages, old-policy ratios, clipping and reference KL.

Symbols and units

J_clip: One PPO surrogate contribution to maximize
reward units
r: Current to rollout-policy action ratio
dimensionless
A: Detached advantage
reward units
epsilon: Surrogate clip width
dimensionless
p,q: Current and reference distributions for this KL direction
probabilities
D_KL: Categorical divergence
nats

For a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL([0.7,0.3] || [0.5,0.5]) is 0.082283 nats, not 0.2 and not the ratio 1.4.

Assumptions One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: −2.4, because all ratios are always forcibly clamped.

Reusing rollouts can reduce sampling demand while increasing sensitivity to stale policies and surrogate choices. A clipped objective does not replace measured policy drift.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Separate sampled advantages, old-policy ratios, clipping and reference KL.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Separate sampled advantages, old-policy ratios, clipping and reference KL.
One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

Interfaces

Inputs and state Input: Retain the identity and action probabilities of the policy that generated each rollout.
Result and check Output: Separate sampled advantages, old-policy ratios, clipping and reference KL.

Check your understanding

For advantage −2 and ratio 1.4, what is the clipped-surrogate contribution?

−2.8, because the minimum retains the worse contribution.
−2.4, because all ratios are always forcibly clamped.
+2.4, because reward signs are ignored.

Answer and reasoning

−2.8, because the minimum retains the worse contribution.

−2.8, because the minimum retains the worse contribution. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

Evidence and scope

Mode: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.