The mechanism
Verifiable reward replaces a learned preference score with an explicit checking procedure. The reward can still be wrong if parsing, tolerance or task specification is wrong. Group-relative methods compare several outputs for the same prompt; the group’s reward distribution provides a baseline and scale. A group in which every output has the same reward provides no relative distinction. That does not establish that the task was solved. R1’s group-relative recipe and Tülu’s PPO-based verifiable-reward recipe are different algorithms, while the lifecycle core uses a smaller REINFORCE bandit.
- Sample a declared number of outputs with a saved policy/RNG identity.
- Preserve raw outputs and verifier decisions, including parse failures.
- Compute a stated group mean and variance convention; handle zero variance explicitly.
- Apply the selected algorithm and score novel cases with an independently specified correctness rule.
Science and assumptions
Verifiers, grouped advantages and RLVR · worked relation
Compute relative group advantages and identify what a verifier does and does not establish.
Symbols and units
- G
- Outputs sampled for one prompt
outputs
- r_i
- Verifier reward
reward units
- bar r
- Group mean
reward units
- s
- Population standard deviation
reward units
- A_i
- Standardized relative advantage
dimensionless
For rewards [1,1,0,0], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are [1,1,−1,−1]. Rewards [0,0,0,0] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.
Assumptions Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.
Engineering example
For rewards [1,1,0,0], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are [1,1,−1,−1]. Rewards [0,0,0,0] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.
Distinctions and tradeoffs
It is a mistake to conclude: All answers must be correct.
Group comparisons reduce dependence on a learned value baseline but can waste batches with no reward variation and amplify verifier mistakes.
Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute relative group advantages and identify what a verifier does and does not establish.
Specifications and interfaces
- Learning mode
- executable checker; worked group normalization
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Compute relative group advantages and identify what a verifier does and does not establish.
Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.
Interfaces
- Inputs and state Input
- Sample a declared number of outputs with a saved policy/RNG identity.
- Result and check Output
- Compute relative group advantages and identify what a verifier does and does not establish.
Check your understanding
What can an all-zero advantage group mean?
- The optimizer has proved the verifier sound.
- Every sampled answer was wrong under the verifier.
- All answers must be correct.
Answer and reasoning
Every sampled answer was wrong under the verifier.
Every sampled answer was wrong under the verifier. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.
Evidence and scope
Mode: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.