# When reward improves but the task gets worse Canonical: https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-posttraining · Atlas v1.3.0 # When reward improves but the task gets worse Compare a rewarded proxy with independently scored task success and locate divergence. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#reward-proxy-overoptimization)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCompare a rewarded proxy with independently scored task success and locate divergence. ## The mechanism An optimization signal defines what updates favor. If it rewards a correlated shortcut rather than the intended task, stronger optimization can increase the proxy while reducing usefulness. This can happen with a learned reward model or a deterministic rule. The lifecycle example makes the error concrete: the faulty verifier rewards every answer a even when the swap target is b. Its reward is fully inspectable, and correctness uses the independent target rule. A reward curve alone therefore cannot serve as the success metric. 1. Predeclare the task score independently of the optimized proxy. 2. Introduce one explicit verifier or rubric defect while preserving the same task cases. 3. Log sampled actions, rewards, baseline/advantage and actual correctness. 4. Compare checkpoints on the same reserved cases, including the slice where the proxy conflicts with the target. ## Science and assumptions ### When reward improves but the task gets worse · worked relation $$ \Delta R_{proxy}>0\quad\not\Rightarrow\quad\Delta S_{task}>0 $$ Compare a rewarded proxy with independently scored task success and locate divergence. Symbols and units R\_proxy Mean optimized reward under its disclosed rule reward units S\_task Independent task success rate dimensionless Delta After-minus-before change at a fixed evaluation boundary difference On four authored questions, policy A gives three correct one-token answers and one wrong one-token answer. A length proxy of 0.1 per token gives mean reward 0.1 and accuracy 3/4. Policy B emits five tokens for every answer but gets only two answers right: reward rises to 0.5 while accuracy falls to 1/2. In the executable binary checker case, always choosing a can receive reward 1 on every prompt but only 1/2 accuracy on balanced a/b targets. Assumptions Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. ## Engineering example On four authored questions, policy A gives three correct one-token answers and one wrong one-token answer. A length proxy of 0.1 per token gives mean reward 0.1 and accuracy 3/4. Policy B emits five tokens for every answer but gets only two answers right: reward rises to 0.5 while accuracy falls to 1/2. In the executable binary checker case, always choosing a can receive reward 1 on every prompt but only 1/2 accuracy on balanced a/b targets. ## Distinctions and tradeoffs It is a mistake to conclude: Higher reward alone. Easy-to-compute proxies provide frequent feedback, while independently scored outcomes may be costly or delayed. Omitting the latter can hide optimization failures. Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compare a rewarded proxy with independently scored task success and locate divergence. ## Specifications and interfaces Learning mode executable faulty verifier; worked length proxy The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Compare a rewarded proxy with independently scored task success and locate divergence. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. ### Interfaces Inputs and state Input Predeclare the task score independently of the optimized proxy. Result and check Output Compare a rewarded proxy with independently scored task success and locate divergence. ## Check your understanding ### Which pair of observations demonstrates divergence? 1. Higher reward alone. 2. A larger number of generated tokens alone. 3. Higher mean length reward together with lower independently scored accuracy. Answer and reasoning Higher mean length reward together with lower independently scored accuracy. Higher mean length reward together with lower independently scored accuracy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. ## Evidence and scope Mode: executable faulty verifier; worked length proxy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: executable faulty verifier; worked length proxy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. ## Sources - [Scaling Laws for Reward Model Overoptimization](https://arxiv.org/html/2210.10760v1) - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/reward-proxy-overoptimization-0c8a90dba139ecf2.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/) - [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/) - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) - [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/) - [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/) - [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)