Learn a reward from pairwise choices

The mechanism

A reward model can assign scalar scores to candidate responses and fit observed pairwise preferences through their score differences. Only differences determine the pairwise probability; adding the same constant to both scores changes neither the prediction nor the loss. The labels can express style, usefulness or a chosen rubric, so a high score is not automatically factual truth. The reward model has its own train/development split and failure modes. It is distinct from the policy that generates answers and from a deterministic checker with an explicitly computable target.

Record prompt, chosen response, rejected response and the rubric that produced the label.
Compute both scores and their difference.
Fit a logistic pairwise loss, retaining gradients for both responses.
Evaluate held-out pairs and inspect order, length and source-support confounders before using scores as rewards.

Science and assumptions

Learn a reward from pairwise choices · worked relation

p(w\succ l)=\sigma(r_w-r_l),\qquad L=-\log\sigma(r_w-r_l),\qquad \frac{\partial L}{\partial(r_w-r_l)}=\sigma(r_w-r_l)-1

Compute a pairwise preference probability and distinguish a learned score from verified correctness.

Symbols and units

r_w,r_l: Chosen and rejected scalar scores
dimensionless
p: Predicted pairwise preference probability
dimensionless
sigma: Logistic function
function
L: Pairwise negative log likelihood
nats per pair

If chosen score minus rejected score is ln3, the chosen probability is 3/(1+3)=0.75 and loss is −ln0.75=0.287682 nats per pair. The derivative with respect to the difference is 0.75−1=−0.25. With learning rate 0.1 applied directly to that scalar difference, it rises by 0.025. Scores [5+ln3,5] yield exactly the same preference probability as [ln3,0].

Assumptions Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Only the absolute scores remain unchanged.

Pairwise judgments can be easier to collect than absolute scores, but reward calibration outside the comparison distribution requires separate evidence.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute a pairwise preference probability and distinguish a learned score from verified correctness.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Compute a pairwise preference probability and distinguish a learned score from verified correctness.
Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.

Interfaces

Inputs and state Input: Record prompt, chosen response, rejected response and the rubric that produced the label.
Result and check Output: Compute a pairwise preference probability and distinguish a learned score from verified correctness.

Check your understanding

What remains unchanged after adding 5 to both response scores?

The preference label must reverse.
The pairwise preference probability and loss.
Only the absolute scores remain unchanged.

Answer and reasoning

The pairwise preference probability and loss.

The pairwise preference probability and loss. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.

Evidence and scope

Mode: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.