{
  "schemaVersion": 1,
  "id": "atlas-lesson-verifiable-reward-group-updates",
  "title": "Verifiers, grouped advantages and RLVR",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compute relative group advantages and identify what a verifier does and does not establish.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/verifiable-reward-group-updates-743dbe4a274025c0.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#verifiable-reward-group-updates",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "a106515d9e050c66d677b5de6582e35b500b9a02bfce2b016b382efc5f5fea6c",
  "contentLength": 6820,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-verifiable-reward-group-updates.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-verifiable-reward-group-updates.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Verifiers, grouped advantages and RLVR",
      "start": 45,
      "end": 463
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 463,
      "end": 1424
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1424,
      "end": 1452
    },
    {
      "id": "section-4",
      "title": "Verifiers, grouped advantages and RLVR · worked relation",
      "start": 1452,
      "end": 2612
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2612,
      "end": 3067
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3067,
      "end": 3490
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3490,
      "end": 3989
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3989,
      "end": 4218
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4218,
      "end": 4247
    },
    {
      "id": "section-10",
      "title": "What can an all-zero advantage group mean?",
      "start": 4247,
      "end": 4750
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4750,
      "end": 5487
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5487,
      "end": 5871
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5871,
      "end": 5894
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5894,
      "end": 6113
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6113,
      "end": 6820
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#verifiable-reward-group-updates"
    },
    {
      "title": "DeepSeek-R1",
      "url": "https://arxiv.org/html/2501.12948v1"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/verifiable-reward-group-updates-743dbe4a274025c0.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    },
    {
      "title": "Frozen base weights and trainable adapters",
      "url": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/"
    },
    {
      "title": "Transfer outputs without confusing size with quality",
      "url": "https://semiaifoundry.com/learn/atlas/teacher-student-distillation/"
    }
  ],
  "recordHash": "35821b03bce6b94b2841f251649cb7b9b296219cac8c551c63e95c2bf075a18b",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Verifiers, grouped advantages and RLVR\n\nCompute relative group advantages and identify what a verifier does and does not establish.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#verifiable-reward-group-updates)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompute relative group advantages and identify what a verifier does and does not establish.\n\n## The mechanism\n\nVerifiable reward replaces a learned preference score with an explicit checking procedure. The reward can still be wrong if parsing, tolerance or task specification is wrong. Group-relative methods compare several outputs for the same prompt; the group’s reward distribution provides a baseline and scale. A group in which every output has the same reward provides no relative distinction. That does not establish that the task was solved. R1’s group-relative recipe and Tülu’s PPO-based verifiable-reward recipe are different algorithms, while the lifecycle core uses a smaller REINFORCE bandit.\n\n1.  Sample a declared number of outputs with a saved policy/RNG identity.\n2.  Preserve raw outputs and verifier decisions, including parse failures.\n3.  Compute a stated group mean and variance convention; handle zero variance explicitly.\n4.  Apply the selected algorithm and score novel cases with an independently specified correctness rule.\n\n## Science and assumptions\n\n### Verifiers, grouped advantages and RLVR · worked relation\n\n$$\n\\bar r=\\frac1G\\sum_i r_i,\\quad s=\\sqrt{\\frac1G\\sum_i(r_i-\\bar r)^2},\\quad A_i=\\begin{cases}(r_i-\\bar r)/s&s>0\\\\0&s=0\\end{cases}\n$$\n\nCompute relative group advantages and identify what a verifier does and does not establish.\n\nSymbols and units\n\nG\n\nOutputs sampled for one prompt\n\noutputs\n\nr\\_i\n\nVerifier reward\n\nreward units\n\nbar r\n\nGroup mean\n\nreward units\n\ns\n\nPopulation standard deviation\n\nreward units\n\nA\\_i\n\nStandardized relative advantage\n\ndimensionless\n\nFor rewards \\[1,1,0,0\\], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are \\[1,1,−1,−1\\]. Rewards \\[0,0,0,0\\] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.\n\nAssumptions Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.\n\n## Engineering example\n\nFor rewards \\[1,1,0,0\\], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are \\[1,1,−1,−1\\]. Rewards \\[0,0,0,0\\] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: All answers must be correct.\n\nGroup comparisons reduce dependence on a learned value baseline but can waste batches with no reward variation and amplify verifier mistakes.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute relative group advantages and identify what a verifier does and does not establish.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable checker; worked group normalization\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompute relative group advantages and identify what a verifier does and does not establish.\n\nExplicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.\n\n### Interfaces\n\nInputs and state Input\n\nSample a declared number of outputs with a saved policy/RNG identity.\n\nResult and check Output\n\nCompute relative group advantages and identify what a verifier does and does not establish.\n\n## Check your understanding\n\n### What can an all-zero advantage group mean?\n\n1.  The optimizer has proved the verifier sound.\n2.  Every sampled answer was wrong under the verifier.\n3.  All answers must be correct.\n\nAnswer and reasoning\n\nEvery sampled answer was wrong under the verifier.\n\nEvery sampled answer was wrong under the verifier. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.\n\n## Evidence and scope\n\nMode: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.\n\n## Sources\n\n-   [DeepSeek-R1](https://arxiv.org/html/2501.12948v1)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/verifiable-reward-group-updates-743dbe4a274025c0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)\n-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)\n-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)"
}