{
  "schemaVersion": 1,
  "id": "atlas-lesson-policy-gradient-kl-control",
  "title": "Rollouts, advantages and PPO constraints",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Separate sampled advantages, old-policy ratios, clipping and reference KL.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/policy-gradient-kl-control-8b31ebf5543b07d9.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#policy-gradient-kl-control",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "32b5181cdb6ae344d4623f4a68edeb2b048e761f661dfdc38c25dbbefb129c0a",
  "contentLength": 6674,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-policy-gradient-kl-control.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-policy-gradient-kl-control.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Rollouts, advantages and PPO constraints",
      "start": 45,
      "end": 426
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 426,
      "end": 1494
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1494,
      "end": 1522
    },
    {
      "id": "section-4",
      "title": "Rollouts, advantages and PPO constraints · worked relation",
      "start": 1522,
      "end": 2695
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2695,
      "end": 3069
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3069,
      "end": 3532
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3532,
      "end": 3939
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3939,
      "end": 4169
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4169,
      "end": 4198
    },
    {
      "id": "section-10",
      "title": "For advantage −2 and ratio 1.4, what is the clipped-surrogate contribution?",
      "start": 4198,
      "end": 4732
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4732,
      "end": 5319
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5319,
      "end": 5728
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5728,
      "end": 5751
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5751,
      "end": 5957
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5957,
      "end": 6674
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#policy-gradient-kl-control"
    },
    {
      "title": "Proximal Policy Optimization Algorithms",
      "url": "https://arxiv.org/pdf/1707.06347"
    },
    {
      "title": "Training language models to follow instructions with human feedback",
      "url": "https://arxiv.org/html/2203.02155v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/policy-gradient-kl-control-8b31ebf5543b07d9.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    },
    {
      "title": "Frozen base weights and trainable adapters",
      "url": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/"
    },
    {
      "title": "Transfer outputs without confusing size with quality",
      "url": "https://semiaifoundry.com/learn/atlas/teacher-student-distillation/"
    }
  ],
  "recordHash": "0b7627f520ba33bfb1a1a658efd724214242297bef5f428af34804cfed9933ed",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Rollouts, advantages and PPO constraints\n\nSeparate sampled advantages, old-policy ratios, clipping and reference KL.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#policy-gradient-kl-control)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveSeparate sampled advantages, old-policy ratios, clipping and reference KL.\n\n## The mechanism\n\nA policy-gradient update learns from actions sampled under a recorded policy. The score-function derivative weights log-probability gradients by an advantage; the sampled action and its assigned reward are treated as data for that update. PPO adds a clipped probability-ratio surrogate for reusing rollout data near the old policy. The old rollout policy and a longer-lived reference policy play different roles. Clipping the surrogate is not a hard guarantee that every new probability ratio or KL stays bounded. The lifecycle core instead implements a two-action REINFORCE update with an exact detached baseline and categorical KL; PPO is worked here only.\n\n1.  Retain the identity and action probabilities of the policy that generated each rollout.\n2.  Compute a stated return/baseline advantage and detach it for the score-function term.\n3.  For PPO, compare current/old ratios with the clipping interval and respect the sign of the advantage.\n4.  Compute reference KL under its explicit distribution and check actual task return independently.\n\n## Science and assumptions\n\n### Rollouts, advantages and PPO constraints · worked relation\n\n$$\nJ_{clip}=\\min(rA,\\operatorname{clip}(r,1-\\epsilon,1+\\epsilon)A),\\quad r=\\frac{\\pi_\\theta(a\\mid s)}{\\pi_{old}(a\\mid s)},\\quad D_{KL}(p\\Vert q)=\\sum_i p_i\\log\\frac{p_i}{q_i}\n$$\n\nSeparate sampled advantages, old-policy ratios, clipping and reference KL.\n\nSymbols and units\n\nJ\\_clip\n\nOne PPO surrogate contribution to maximize\n\nreward units\n\nr\n\nCurrent to rollout-policy action ratio\n\ndimensionless\n\nA\n\nDetached advantage\n\nreward units\n\nepsilon\n\nSurrogate clip width\n\ndimensionless\n\np,q\n\nCurrent and reference distributions for this KL direction\n\nprobabilities\n\nD\\_KL\n\nCategorical divergence\n\nnats\n\nFor a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL(\\[0.7,0.3\\] || \\[0.5,0.5\\]) is 0.082283 nats, not 0.2 and not the ratio 1.4.\n\nAssumptions One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.\n\n## Engineering example\n\nFor a sampled action with old probability 0.5, current probability 0.7 and advantage +2, ratio is 1.4. With epsilon 0.2, PPO’s contribution is min(1.4×2,1.2×2)=2.4. For advantage −2 it is min(−2.8,−2.4)=−2.8. The sign determines which incentive is clipped. Separately, KL(\\[0.7,0.3\\] || \\[0.5,0.5\\]) is 0.082283 nats, not 0.2 and not the ratio 1.4.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: −2.4, because all ratios are always forcibly clamped.\n\nReusing rollouts can reduce sampling demand while increasing sensitivity to stale policies and surrogate choices. A clipped objective does not replace measured policy drift.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Separate sampled advantages, old-policy ratios, clipping and reference KL.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nSeparate sampled advantages, old-policy ratios, clipping and reference KL.\n\nOne sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.\n\n### Interfaces\n\nInputs and state Input\n\nRetain the identity and action probabilities of the policy that generated each rollout.\n\nResult and check Output\n\nSeparate sampled advantages, old-policy ratios, clipping and reference KL.\n\n## Check your understanding\n\n### For advantage −2 and ratio 1.4, what is the clipped-surrogate contribution?\n\n1.  −2.8, because the minimum retains the worse contribution.\n2.  −2.4, because all ratios are always forcibly clamped.\n3.  +2.4, because reward signs are ignored.\n\nAnswer and reasoning\n\n−2.8, because the minimum retains the worse contribution.\n\n−2.8, because the minimum retains the worse contribution. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.\n\n## Evidence and scope\n\nMode: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. One sampled-action term with fixed advantage. PPO formula is worked-only; executable core uses REINFORCE plus exact two-action KL, not PPO or GRPO.\n\n## Sources\n\n-   [Proximal Policy Optimization Algorithms](https://arxiv.org/pdf/1707.06347)\n-   [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/policy-gradient-kl-control-8b31ebf5543b07d9.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n\n### Related lessons\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)\n-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)\n-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)"
}