{
  "schemaVersion": 1,
  "id": "atlas-lesson-preference-reward-model",
  "title": "Learn a reward from pairwise choices",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compute a pairwise preference probability and distinguish a learned score from verified correctness.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/preference-reward-model/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/preference-reward-model-c9a7a1809b81ee74.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#preference-reward-model",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "dc22b3da3f78ef26274c3b7469524035b388cadd22f09905fcb745fe0f24ec2d",
  "contentLength": 6595,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-preference-reward-model.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-preference-reward-model.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Learn a reward from pairwise choices",
      "start": 45,
      "end": 471
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 471,
      "end": 1404
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1404,
      "end": 1432
    },
    {
      "id": "section-4",
      "title": "Learn a reward from pairwise choices · worked relation",
      "start": 1432,
      "end": 2507
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2507,
      "end": 2888
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2888,
      "end": 3345
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3345,
      "end": 3817
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3817,
      "end": 4075
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4075,
      "end": 4104
    },
    {
      "id": "section-10",
      "title": "What remains unchanged after adding 5 to both response scores?",
      "start": 4104,
      "end": 4612
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4612,
      "end": 5277
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5277,
      "end": 5677
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5677,
      "end": 5700
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5700,
      "end": 5892
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5892,
      "end": 6595
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#preference-reward-model"
    },
    {
      "title": "Training language models to follow instructions with human feedback",
      "url": "https://arxiv.org/html/2203.02155v1"
    },
    {
      "title": "Direct Preference Optimization",
      "url": "https://arxiv.org/html/2305.18290v3"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/preference-reward-model-c9a7a1809b81ee74.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Post-training & feedback",
      "url": "https://semiaifoundry.com/learn/atlas/post-training/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "DPO policy/reference likelihood ratios",
      "url": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    },
    {
      "title": "Frozen base weights and trainable adapters",
      "url": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/"
    }
  ],
  "recordHash": "5057d5ed7f5431f95163b4634fd2da2106a9f485ab3ec4f2b919bd0c96dbd037",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# Learn a reward from pairwise choices\n\nCompute a pairwise preference probability and distinguish a learned score from verified correctness.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#preference-reward-model)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompute a pairwise preference probability and distinguish a learned score from verified correctness.\n\n## The mechanism\n\nA reward model can assign scalar scores to candidate responses and fit observed pairwise preferences through their score differences. Only differences determine the pairwise probability; adding the same constant to both scores changes neither the prediction nor the loss. The labels can express style, usefulness or a chosen rubric, so a high score is not automatically factual truth. The reward model has its own train/development split and failure modes. It is distinct from the policy that generates answers and from a deterministic checker with an explicitly computable target.\n\n1.  Record prompt, chosen response, rejected response and the rubric that produced the label.\n2.  Compute both scores and their difference.\n3.  Fit a logistic pairwise loss, retaining gradients for both responses.\n4.  Evaluate held-out pairs and inspect order, length and source-support confounders before using scores as rewards.\n\n## Science and assumptions\n\n### Learn a reward from pairwise choices · worked relation\n\n$$\np(w\\succ l)=\\sigma(r_w-r_l),\\qquad L=-\\log\\sigma(r_w-r_l),\\qquad \\frac{\\partial L}{\\partial(r_w-r_l)}=\\sigma(r_w-r_l)-1\n$$\n\nCompute a pairwise preference probability and distinguish a learned score from verified correctness.\n\nSymbols and units\n\nr\\_w,r\\_l\n\nChosen and rejected scalar scores\n\ndimensionless\n\np\n\nPredicted pairwise preference probability\n\ndimensionless\n\nsigma\n\nLogistic function\n\nfunction\n\nL\n\nPairwise negative log likelihood\n\nnats per pair\n\nIf chosen score minus rejected score is ln3, the chosen probability is 3/(1+3)=0.75 and loss is −ln0.75=0.287682 nats per pair. The derivative with respect to the difference is 0.75−1=−0.25. With learning rate 0.1 applied directly to that scalar difference, it rises by 0.025. Scores \\[5+ln3,5\\] yield exactly the same preference probability as \\[ln3,0\\].\n\nAssumptions Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.\n\n## Engineering example\n\nIf chosen score minus rejected score is ln3, the chosen probability is 3/(1+3)=0.75 and loss is −ln0.75=0.287682 nats per pair. The derivative with respect to the difference is 0.75−1=−0.25. With learning rate 0.1 applied directly to that scalar difference, it rises by 0.025. Scores \\[5+ln3,5\\] yield exactly the same preference probability as \\[ln3,0\\].\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Only the absolute scores remain unchanged.\n\nPairwise judgments can be easier to collect than absolute scores, but reward calibration outside the comparison distribution requires separate evidence.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute a pairwise preference probability and distinguish a learned score from verified correctness.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompute a pairwise preference probability and distinguish a learned score from verified correctness.\n\nBradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.\n\n### Interfaces\n\nInputs and state Input\n\nRecord prompt, chosen response, rejected response and the rubric that produced the label.\n\nResult and check Output\n\nCompute a pairwise preference probability and distinguish a learned score from verified correctness.\n\n## Check your understanding\n\n### What remains unchanged after adding 5 to both response scores?\n\n1.  The preference label must reverse.\n2.  The pairwise preference probability and loss.\n3.  Only the absolute scores remain unchanged.\n\nAnswer and reasoning\n\nThe pairwise preference probability and loss.\n\nThe pairwise preference probability and loss. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.\n\n## Evidence and scope\n\nMode: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Bradley–Terry logistic score-difference model with one explicit preference label. No learned reward model is implemented in the lifecycle core; its reward stage uses a disclosed checker.\n\n## Sources\n\n-   [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1)\n-   [Direct Preference Optimization](https://arxiv.org/html/2305.18290v3)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/preference-reward-model-c9a7a1809b81ee74.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Post-training & feedback](https://semiaifoundry.com/learn/atlas/post-training/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n\n### Related lessons\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [DPO policy/reference likelihood ratios](https://semiaifoundry.com/learn/atlas/direct-preference-objective/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)\n-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)"
}