{"id":"direct-preference-objective","name":"DPO policy/reference likelihood ratios","shortName":"DPO policy/reference likelihood ratios","role":"Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.","description":"DPO changes the policy directly using chosen and rejected response likelihoods relative to a reference. The reference supplies a fixed baseline for the ratio; it must not silently follow each policy update. Response log probability normally sums the included autoregressive token log probabilities. Pair averaging and token averaging are different objective choices. Preference labels specify which answer to favor, but they do not prove truth or guarantee that the highest-probability token in the full vocabulary becomes correct. The lifecycle implementation exposes the actual policy/reference scores and independent task accuracy.","mechanism":["Freeze a reference checkpoint at the declared branch point.","Score chosen and rejected responses under both models with the same response-token convention.","Form the difference of log-ratios, scale by beta and apply the negative log-sigmoid loss.","Backpropagate only through policy scores; evaluate held-out response behavior separately."],"learningObjective":"Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.","engineeringExample":"For an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291.","evidenceNotes":"Mode: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable DPO","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.","context":"Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Freeze a reference checkpoint at the declared branch point."},{"name":"Result and check","direction":"out","signal":"Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator."}],"tradeoff":"DPO avoids fitting and sampling from a separate reward model in this update, but remains dependent on the preference data and reference choice.","whyAI":"Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.","misconception":"It is a mistake to conclude: Both policy and reference follow the same update.","check":{"question":"Which state receives gradients in the declared DPO branch?","options":["Both policy and reference follow the same update.","Only the reference is trained.","The current policy; the branch reference remains fixed."],"answer":2,"explanation":"The current policy; the branch reference remains fixed. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels."},"prerequisites":["sft-chat-loss-masks","preference-reward-model"],"related":["sft-chat-loss-masks","preference-reward-model","policy-gradient-kl-control","verifiable-reward-group-updates","reward-proxy-overoptimization","low-rank-adaptation","teacher-student-distillation"],"sources":[{"title":"Direct Preference Optimization","url":"https://arxiv.org/html/2305.18290v3"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-posttraining","subbranch":"DPO policy/reference likelihood ratios","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.","science":[{"title":"DPO policy/reference likelihood ratios · worked relation","equation":"L_{DPO}=-\\log\\sigma\\!\\left(\\beta\\left[\\log\\frac{\\pi_\\theta(y_w\\mid x)}{\\pi_{ref}(y_w\\mid x)}-\\log\\frac{\\pi_\\theta(y_l\\mid x)}{\\pi_{ref}(y_l\\mid x)}\\right]\\right)","explanation":"For an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291.","assumptions":"Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels."}],"history":[]}