The mechanism
An optimization signal defines what updates favor. If it rewards a correlated shortcut rather than the intended task, stronger optimization can increase the proxy while reducing usefulness. This can happen with a learned reward model or a deterministic rule. The lifecycle example makes the error concrete: the faulty verifier rewards every answer a even when the swap target is b. Its reward is fully inspectable, and correctness uses the independent target rule. A reward curve alone therefore cannot serve as the success metric.
- Predeclare the task score independently of the optimized proxy.
- Introduce one explicit verifier or rubric defect while preserving the same task cases.
- Log sampled actions, rewards, baseline/advantage and actual correctness.
- Compare checkpoints on the same reserved cases, including the slice where the proxy conflicts with the target.
Science and assumptions
When reward improves but the task gets worse · worked relation
Compare a rewarded proxy with independently scored task success and locate divergence.
Symbols and units
- R_proxy
- Mean optimized reward under its disclosed rule
reward units
- S_task
- Independent task success rate
dimensionless
- Delta
- After-minus-before change at a fixed evaluation boundary
difference
On four authored questions, policy A gives three correct one-token answers and one wrong one-token answer. A length proxy of 0.1 per token gives mean reward 0.1 and accuracy 3/4. Policy B emits five tokens for every answer but gets only two answers right: reward rises to 0.5 while accuracy falls to 1/2. In the executable binary checker case, always choosing a can receive reward 1 on every prompt but only 1/2 accuracy on balanced a/b targets.
Assumptions Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory.
Engineering example
On four authored questions, policy A gives three correct one-token answers and one wrong one-token answer. A length proxy of 0.1 per token gives mean reward 0.1 and accuracy 3/4. Policy B emits five tokens for every answer but gets only two answers right: reward rises to 0.5 while accuracy falls to 1/2. In the executable binary checker case, always choosing a can receive reward 1 on every prompt but only 1/2 accuracy on balanced a/b targets.
Distinctions and tradeoffs
It is a mistake to conclude: Higher reward alone.
Easy-to-compute proxies provide frequent feedback, while independently scored outcomes may be costly or delayed. Omitting the latter can hide optimization failures.
Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compare a rewarded proxy with independently scored task success and locate divergence.
Specifications and interfaces
- Learning mode
- executable faulty verifier; worked length proxy
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Compare a rewarded proxy with independently scored task success and locate divergence.
Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory.
Interfaces
- Inputs and state Input
- Predeclare the task score independently of the optimized proxy.
- Result and check Output
- Compare a rewarded proxy with independently scored task success and locate divergence.
Check your understanding
Which pair of observations demonstrates divergence?
- Higher reward alone.
- A larger number of generated tokens alone.
- Higher mean length reward together with lower independently scored accuracy.
Answer and reasoning
Higher mean length reward together with lower independently scored accuracy.
Higher mean length reward together with lower independently scored accuracy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory.
Evidence and scope
Mode: executable faulty verifier; worked length proxy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: executable faulty verifier; worked length proxy. Authored counterexample and a separately specified task scorer. No claim that every reward model inevitably fails or that a particular real system followed this trajectory.