{"id":"preference-reward-model","name":"Learn a reward from pairwise choices","shortName":"Learn a reward from pairwise choices","role":"Compute a pairwise preference probability and distinguish a learned score from verified correctness.","description":"A reward model can assign scalar scores to candidate responses and fit observed pairwise preferences through their score differences. Only differences determine the pairwise probability; adding the same constant to both scores changes neither the prediction nor the loss. The labels can express style, usefulness or a chosen rubric, so a high score is not automatically factual truth. The reward model has its own train/development split and failure modes. It is distinct from the policy that generates answers and from a deterministic checker with an explicitly computable target.","mechanism":["Record prompt, chosen response, rejected response and the rubric that produced the label.","Compute both scores and their difference.","Fit a logistic pairwise loss, retaining gradients for both responses.","Evaluate held-out pairs and inspect order, length and source-support confounders before using scores as rewards."],"learningObjective":"Compute a pairwise preference probability and distinguish a learned score from verified correctness.","engineeringExample":"If chosen score minus rejected score is ln3, the chosen probability is 3/(1+3)=0.75 and loss is −ln0.75=0.287682 nats per pair. The derivative with respect to the difference is 0.75−1=−0.25. With learning rate 0.1 applied directly to that scalar difference, it rises by 0.025. Scores [5+ln3,5] yield exactly the same preference probability as [ln3,0].","evidenceNotes":"Mode: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compute a pairwise preference probability and distinguish a learned score from verified correctness.","context":"Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Record prompt, chosen response, rejected response and the rubric that produced the label."},{"name":"Result and check","direction":"out","signal":"Compute a pairwise preference probability and distinguish a learned score from verified correctness."}],"tradeoff":"Pairwise judgments can be easier to collect than absolute scores, but reward calibration outside the comparison distribution requires separate evidence.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute a pairwise preference probability and distinguish a learned score from verified correctness.","misconception":"It is a mistake to conclude: Only the absolute scores remain unchanged.","check":{"question":"What remains unchanged after adding 5 to both response scores?","options":["The preference label must reverse.","The pairwise preference probability and loss.","Only the absolute scores remain unchanged."],"answer":1,"explanation":"The pairwise preference probability and loss. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker."},"prerequisites":["post-training","loss-optimizer"],"related":["post-training","loss-optimizer","sft-chat-loss-masks","direct-preference-objective","policy-gradient-kl-control","verifiable-reward-group-updates","reward-proxy-overoptimization","low-rank-adaptation"],"sources":[{"title":"Training language models to follow instructions with human feedback","url":"https://arxiv.org/html/2203.02155v1"},{"title":"Direct Preference Optimization","url":"https://arxiv.org/html/2305.18290v3"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"Learn a reward from pairwise choices","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.","science":[{"title":"Learn a reward from pairwise choices · worked relation","equation":"p(w\\succ l)=\\sigma(r_w-r_l),\\qquad L=-\\log\\sigma(r_w-r_l),\\qquad \\frac{\\partial L}{\\partial(r_w-r_l)}=\\sigma(r_w-r_l)-1","explanation":"If chosen score minus rejected score is ln3, the chosen probability is 3/(1+3)=0.75 and loss is −ln0.75=0.287682 nats per pair. The derivative with respect to the difference is 0.75−1=−0.25. With learning rate 0.1 applied directly to that scalar difference, it rises by 0.025. Scores [5+ln3,5] yield exactly the same preference probability as [ln3,0].","assumptions":"Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker."}],"history":[]}