# Rollouts, advantages and PPO constraints

Canonical: https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

model-lifecycle-posttraining · Atlas v1.3.0

# Rollouts, advantages and PPO constraints

Separate sampled advantages, old-policy ratios, clipping and reference KL.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#policy-gradient-kl-control)Mathematical, data and behavior space; logical dimensions are not physical distance

Learning objectiveSeparate sampled advantages, old-policy ratios, clipping and reference KL.

## The mechanism

A policy-gradient update learns from actions sampled under a recorded policy. The score-function derivative weights log-probability gradients by an advantage; the sampled action and its assigned reward are treated as data for that update. PPO adds a clipped probability-ratio surrogate for reusing rollout data near the old policy. The old rollout policy and a longer-lived reference policy play different roles. Clipping the surrogate is not a hard guarantee that every new probability ratio or KL stays bounded. The lifecycle core instead implements a two-action REINFORCE update with an exact detached baseline and categorical KL; PPO is worked here only.

1.  Retain the identity and action probabilities of the policy that generated each rollout.
2.  Compute a stated return/baseline advantage and detach it for the score-function term.
3.  For PPO, compare current/old ratios with the clipping interval and respect the sign of the advantage.
4.  Compute reference KL under its explicit distribution and check actual task return independently.

## Science and assumptions

### Rollouts, advantages and PPO constraints · worked relation

$$
J_{clip}=\min(rA,\operatorname{clip}(r,1-\epsilon,1+\epsilon)A),\quad r=\frac{\pi_\theta(a\mid s)}{\pi_{old}(a\mid s)},\quad D_{KL}(p\Vert q)=\sum_i p_i\log\frac{p_i}{q_i}
$$

Separate sampled advantages, old-policy ratios, clipping and reference KL.

Symbols and units

J\_clip

One PPO surrogate contribution to maximize

reward units

r

Current to rollout-policy action ratio

dimensionless

A

Detached advantage

reward units

epsilon

Surrogate clip width

dimensionless

p,q

Current and reference distributions for this KL direction

probabilities

D\_KL

Categorical divergence

nats

For a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL(\[0.7,0.3\] || \[0.5,0.5\]) is 0.082283 nats, not 0.2 and not the ratio 1.4.

Assumptions One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

## Engineering example

For a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL(\[0.7,0.3\] || \[0.5,0.5\]) is 0.082283 nats, not 0.2 and not the ratio 1.4.

## Distinctions and tradeoffs

It is a mistake to conclude: −2.4, because all ratios are always forcibly clamped.

Reusing rollouts can reduce sampling demand while increasing sensitivity to stale policies and surrogate choices. A clipped objective does not replace measured policy drift.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Separate sampled advantages, old-policy ratios, clipping and reference KL.

## Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Separate sampled advantages, old-policy ratios, clipping and reference KL.

One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

### Interfaces

Inputs and state Input

Retain the identity and action probabilities of the policy that generated each rollout.

Result and check Output

Separate sampled advantages, old-policy ratios, clipping and reference KL.

## Check your understanding

### For advantage −2 and ratio 1.4, what is the clipped-surrogate contribution?

1.  −2.8, because the minimum retains the worse contribution.
2.  −2.4, because all ratios are always forcibly clamped.
3.  +2.4, because reward signs are ignored.

Answer and reasoning

−2.8, because the minimum retains the worse contribution.

−2.8, because the minimum retains the worse contribution. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

## Evidence and scope

Mode: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.

## Sources

-   [Proximal Policy Optimization Algorithms](https://arxiv.org/pdf/1707.06347)
-   [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/policy-gradient-kl-control-8b31ebf5543b07d9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)
-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)

### Related lessons

-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)
-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)
-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)
-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)
-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)
-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)