# Verifiers, grouped advantages and RLVR

Canonical: https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

model-lifecycle-posttraining · Atlas v1.3.0

# Verifiers, grouped advantages and RLVR

Compute relative group advantages and identify what a verifier does and does not establish.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#verifiable-reward-group-updates)Mathematical, data and behavior space; logical dimensions are not physical distance

Learning objectiveCompute relative group advantages and identify what a verifier does and does not establish.

## The mechanism

Verifiable reward replaces a learned preference score with an explicit checking procedure. The reward can still be wrong if parsing, tolerance or task specification is wrong. Group-relative methods compare several outputs for the same prompt; the group’s reward distribution provides a baseline and scale. A group in which every output has the same reward provides no relative distinction. That does not establish that the task was solved. R1’s group-relative recipe and Tülu’s PPO-based verifiable-reward recipe are different algorithms, while the lifecycle core uses a smaller REINFORCE bandit.

1.  Sample a declared number of outputs with a saved policy/RNG identity.
2.  Preserve raw outputs and verifier decisions, including parse failures.
3.  Compute a stated group mean and variance convention; handle zero variance explicitly.
4.  Apply the selected algorithm and score novel cases with an independently specified correctness rule.

## Science and assumptions

### Verifiers, grouped advantages and RLVR · worked relation

$$
\bar r=\frac1G\sum_i r_i,\quad s=\sqrt{\frac1G\sum_i(r_i-\bar r)^2},\quad A_i=\begin{cases}(r_i-\bar r)/s&s>0\\0&s=0\end{cases}
$$

Compute relative group advantages and identify what a verifier does and does not establish.

Symbols and units

G

Outputs sampled for one prompt

outputs

r\_i

Verifier reward

reward units

bar r

Group mean

reward units

s

Population standard deviation

reward units

A\_i

Standardized relative advantage

dimensionless

For rewards \[1,1,0,0\], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are \[1,1,−1,−1\]. Rewards \[0,0,0,0\] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.

Assumptions Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.

## Engineering example

For rewards \[1,1,0,0\], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are \[1,1,−1,−1\]. Rewards \[0,0,0,0\] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.

## Distinctions and tradeoffs

It is a mistake to conclude: All answers must be correct.

Group comparisons reduce dependence on a learned value baseline but can waste batches with no reward variation and amplify verifier mistakes.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute relative group advantages and identify what a verifier does and does not establish.

## Specifications and interfaces

Learning mode

executable checker; worked group normalization

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Compute relative group advantages and identify what a verifier does and does not establish.

Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.

### Interfaces

Inputs and state Input

Sample a declared number of outputs with a saved policy/RNG identity.

Result and check Output

Compute relative group advantages and identify what a verifier does and does not establish.

## Check your understanding

### What can an all-zero advantage group mean?

1.  The optimizer has proved the verifier sound.
2.  Every sampled answer was wrong under the verifier.
3.  All answers must be correct.

Answer and reasoning

Every sampled answer was wrong under the verifier.

Every sampled answer was wrong under the verifier. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.

## Evidence and scope

Mode: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.

## Sources

-   [DeepSeek-R1](https://arxiv.org/html/2501.12948v1)
-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/verifiable-reward-group-updates-743dbe4a274025c0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)
-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)

### Related lessons

-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)
-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)
-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)
-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)
-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)
-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)