{"id":"verifiable-reward-group-updates","name":"Verifiers, grouped advantages and RLVR","shortName":"Verifiers, grouped advantages and RLVR","role":"Compute relative group advantages and identify what a verifier does and does not establish.","description":"Verifiable reward replaces a learned preference score with an explicit checking procedure. The reward can still be wrong if parsing, tolerance or task specification is wrong. Group-relative methods compare several outputs for the same prompt; the group’s reward distribution provides a baseline and scale. A group in which every output has the same reward provides no relative distinction. That does not establish that the task was solved. R1’s group-relative recipe and Tülu’s PPO-based verifiable-reward recipe are different algorithms, while the lifecycle core uses a smaller REINFORCE bandit.","mechanism":["Sample a declared number of outputs with a saved policy/RNG identity.","Preserve raw outputs and verifier decisions, including parse failures.","Compute a stated group mean and variance convention; handle zero variance explicitly.","Apply the selected algorithm and score novel cases with an independently specified correctness rule."],"learningObjective":"Compute relative group advantages and identify what a verifier does and does not establish.","engineeringExample":"For rewards [1,1,0,0], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are [1,1,−1,−1]. Rewards [0,0,0,0] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.","evidenceNotes":"Mode: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable checker; worked group normalization","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compute relative group advantages and identify what a verifier does and does not establish.","context":"Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Sample a declared number of outputs with a saved policy/RNG identity."},{"name":"Result and check","direction":"out","signal":"Compute relative group advantages and identify what a verifier does and does not establish."}],"tradeoff":"Group comparisons reduce dependence on a learned value baseline but can waste batches with no reward variation and amplify verifier mistakes.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute relative group advantages and identify what a verifier does and does not establish.","misconception":"It is a mistake to conclude: All answers must be correct.","check":{"question":"What can an all-zero advantage group mean?","options":["The optimizer has proved the verifier sound.","Every sampled answer was wrong under the verifier.","All answers must be correct."],"answer":1,"explanation":"Every sampled answer was wrong under the verifier. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator."},"prerequisites":["policy-gradient-kl-control","evaluation"],"related":["policy-gradient-kl-control","evaluation","sft-chat-loss-masks","preference-reward-model","direct-preference-objective","reward-proxy-overoptimization","low-rank-adaptation","teacher-student-distillation"],"sources":[{"title":"DeepSeek-R1","url":"https://arxiv.org/html/2501.12948v1"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"Verifiers, grouped advantages and RLVR","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable checker; worked group normalization. Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator.","science":[{"title":"Verifiers, grouped advantages and RLVR · worked relation","equation":"\\bar r=\\frac1G\\sum_i r_i,\\quad s=\\sqrt{\\frac1G\\sum_i(r_i-\\bar r)^2},\\quad A_i=\\begin{cases}(r_i-\\bar r)/s&s>0\\\\0&s=0\\end{cases}","explanation":"For rewards [1,1,0,0], mean is 0.5 and population standard deviation is 0.5. Standardized advantages are [1,1,−1,−1]. Rewards [0,0,0,0] have zero variance; this worked convention returns zero advantages rather than dividing by zero. All four outputs can therefore be wrong while producing no within-group signal. The studio’s four sampled actions per prompt use an exact policy baseline instead of this group normalization.","assumptions":"Explicit population-variance and zero-variance teaching convention; real implementations may add epsilon or use another normalization. This group arithmetic is not the core bandit estimator."}],"history":[]}