# DPO policy/reference likelihood ratios Canonical: https://semiaifoundry.com/learn/atlas/direct-preference-objective/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-posttraining · Atlas v1.3.0 # DPO policy/reference likelihood ratios Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#direct-preference-objective)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCompute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. ## The mechanism DPO changes the policy directly using chosen and rejected response likelihoods relative to a reference. The reference supplies a fixed baseline for the ratio; it must not silently follow each policy update. Response log probability normally sums the included autoregressive token log probabilities. Pair averaging and token averaging are different objective choices. Preference labels specify which answer to favor, but they do not prove truth or guarantee that the highest-probability token in the full vocabulary becomes correct. The lifecycle implementation exposes the actual policy/reference scores and independent task accuracy. 1. Freeze a reference checkpoint at the declared branch point. 2. Score chosen and rejected responses under both models with the same response-token convention. 3. Form the difference of log-ratios, scale by beta and apply the negative log-sigmoid loss. 4. Backpropagate only through policy scores; evaluate held-out response behavior separately. ## Science and assumptions ### DPO policy/reference likelihood ratios · worked relation $$ L_{DPO}=-\log\sigma\!\left(\beta\left[\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}\right]\right) $$ Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. Symbols and units L\_DPO Loss for one labeled preference pair nats per pair pi\_theta Current policy response probability dimensionless pi\_ref Frozen reference response probability dimensionless y\_w,y\_l Chosen and rejected responses token sequences x Prompt token sequence beta Log-ratio scale dimensionless For an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291. Assumptions Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. ## Engineering example For an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291. ## Distinctions and tradeoffs It is a mistake to conclude: Both policy and reference follow the same update. DPO avoids fitting and sampling from a separate reward model in this update, but remains dependent on the preference data and reference choice. Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. ## Specifications and interfaces Learning mode executable DPO The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. ### Interfaces Inputs and state Input Freeze a reference checkpoint at the declared branch point. Result and check Output Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator. ## Check your understanding ### Which state receives gradients in the declared DPO branch? 1. Both policy and reference follow the same update. 2. Only the reference is trained. 3. The current policy; the branch reference remains fixed. Answer and reasoning The current policy; the branch reference remains fixed. The current policy; the branch reference remains fixed. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. ## Evidence and scope Mode: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. ## Sources - [Direct Preference Optimization](https://arxiv.org/html/2305.18290v3) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/direct-preference-objective-ec338a46ac48d11f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/) - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) ### Related lessons - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) - [Verifiers, grouped advantages and RLVR](https://semiaifoundry.com/learn/atlas/verifiable-reward-group-updates/) - [When reward improves but the task gets worse](https://semiaifoundry.com/learn/atlas/reward-proxy-overoptimization/) - [Frozen base weights and trainable adapters](https://semiaifoundry.com/learn/atlas/low-rank-adaptation/) - [Transfer outputs without confusing size with quality](https://semiaifoundry.com/learn/atlas/teacher-student-distillation/)