{
  "schemaVersion": 1,
  "id": "atlas-lesson-direct-preference-objective",
  "title": "DPO policy/reference likelihood ratios",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/direct-preference-objective/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-posttraining",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/direct-preference-objective-ec338a46ac48d11f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#direct-preference-objective",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "95059fe768b375077fb0b3400152e9581761020b10d4ab1e36bfcb2870a526d1",
  "contentLength": 6729,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-direct-preference-objective.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-direct-preference-objective.md",
  "sections": [
    {
      "id": "section-1",
      "title": "DPO policy/reference likelihood ratios",
      "start": 45,
      "end": 477
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 477,
      "end": 1483
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1483,
      "end": 1511
    },
    {
      "id": "section-4",
      "title": "DPO policy/reference likelihood ratios · worked relation",
      "start": 1511,
      "end": 2791
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2791,
      "end": 3248
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3248,
      "end": 3703
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3703,
      "end": 4159
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4159,
      "end": 4387
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4387,
      "end": 4416
    },
    {
      "id": "section-10",
      "title": "Which state receives gradients in the declared DPO branch?",
      "start": 4416,
      "end": 4937
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4937,
      "end": 5570
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5570,
      "end": 5863
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5863,
      "end": 5886
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5886,
      "end": 6120
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6120,
      "end": 6729
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#direct-preference-objective"
    },
    {
      "title": "Direct Preference Optimization",
      "url": "https://arxiv.org/html/2305.18290v3"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/direct-preference-objective-ec338a46ac48d11f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Verifiers, grouped advantages and RLVR",
      "url": "https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/"
    },
    {
      "title": "When reward improves but the task gets worse",
      "url": "https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/"
    },
    {
      "title": "Frozen base weights and trainable adapters",
      "url": "https://semiaifoundry.com/learn/atlas/low-rank-adaptation/"
    },
    {
      "title": "Transfer outputs without confusing size with quality",
      "url": "https://semiaifoundry.com/learn/atlas/teacher-student-distillation/"
    }
  ],
  "recordHash": "c0fb57de11b90e1aa7828235161814afb52bb280230e93d1cf47590efe023ef0",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-posttraining · Atlas v1.3.0\n\n# DPO policy/reference likelihood ratios\n\nCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#direct-preference-objective)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\n## The mechanism\n\nDPO changes the policy directly using chosen and rejected response likelihoods relative to a reference. The reference supplies a fixed baseline for the ratio; it must not silently follow each policy update. Response log probability normally sums the included autoregressive token log probabilities. Pair averaging and token averaging are different objective choices. Preference labels specify which answer to favor, but they do not prove truth or guarantee that the highest-probability token in the full vocabulary becomes correct. The lifecycle implementation exposes the actual policy/reference scores and independent task accuracy.\n\n1.  Freeze a reference checkpoint at the declared branch point.\n2.  Score chosen and rejected responses under both models with the same response-token convention.\n3.  Form the difference of log-ratios, scale by beta and apply the negative log-sigmoid loss.\n4.  Backpropagate only through policy scores; evaluate held-out response behavior separately.\n\n## Science and assumptions\n\n### DPO policy/reference likelihood ratios · worked relation\n\n$$\nL_{DPO}=-\\log\\sigma\\!\\left(\\beta\\left[\\log\\frac{\\pi_\\theta(y_w\\mid x)}{\\pi_{ref}(y_w\\mid x)}-\\log\\frac{\\pi_\\theta(y_l\\mid x)}{\\pi_{ref}(y_l\\mid x)}\\right]\\right)\n$$\n\nCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\nSymbols and units\n\nL\\_DPO\n\nLoss for one labeled preference pair\n\nnats per pair\n\npi\\_theta\n\nCurrent policy response probability\n\ndimensionless\n\npi\\_ref\n\nFrozen reference response probability\n\ndimensionless\n\ny\\_w,y\\_l\n\nChosen and rejected responses\n\ntoken sequences\n\nx\n\nPrompt\n\ntoken sequence\n\nbeta\n\nLog-ratio scale\n\ndimensionless\n\nFor an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291.\n\nAssumptions Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.\n\n## Engineering example\n\nFor an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Both policy and reference follow the same update.\n\nDPO avoids fitting and sampling from a separate reward model in this update, but remains dependent on the preference data and reference choice.\n\nSeparate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\n## Specifications and interfaces\n\nLearning mode\n\nexecutable DPO\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\nPositive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.\n\n### Interfaces\n\nInputs and state Input\n\nFreeze a reference checkpoint at the declared branch point.\n\nResult and check Output\n\nCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.\n\n## Check your understanding\n\n### Which state receives gradients in the declared DPO branch?\n\n1.  Both policy and reference follow the same update.\n2.  Only the reference is trained.\n3.  The current policy; the branch reference remains fixed.\n\nAnswer and reasoning\n\nThe current policy; the branch reference remains fixed.\n\nThe current policy; the branch reference remains fixed. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.\n\n## Evidence and scope\n\nMode: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.\n\n## Sources\n\n-   [Direct Preference Optimization](https://arxiv.org/html/2305.18290v3)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/direct-preference-objective-ec338a46ac48d11f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n\n### Related lessons\n\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n-   [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/)\n-   [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/)\n-   [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/)\n-   [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)"
}