DPO policy/reference likelihood ratios

The mechanism

DPO changes the policy directly using chosen and rejected response likelihoods relative to a reference. The reference supplies a fixed baseline for the ratio; it must not silently follow each policy update. Response log probability normally sums the included autoregressive token log probabilities. Pair averaging and token averaging are different objective choices. Preference labels specify which answer to favor, but they do not prove truth or guarantee that the highest-probability token in the full vocabulary becomes correct. The lifecycle implementation exposes the actual policy/reference scores and independent task accuracy.

Freeze a reference checkpoint at the declared branch point.
Score chosen and rejected responses under both models with the same response-token convention.
Form the difference of log-ratios, scale by beta and apply the negative log-sigmoid loss.
Backpropagate only through policy scores; evaluate held-out response behavior separately.

Science and assumptions

DPO policy/reference likelihood ratios · worked relation

L_{DPO}=-\log\sigma\!\left(\beta\left[\log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}-\log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)}\right]\right)

Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.

Symbols and units

L_DPO: Loss for one labeled preference pair
nats per pair
pi_theta: Current policy response probability
dimensionless
pi_ref: Frozen reference response probability
dimensionless
y_w,y_l: Chosen and rejected responses
token sequences
x: Prompt
token sequence
beta: Log-ratio scale
dimensionless

For an authored two-response case, policy probabilities are chosen 0.6 and rejected 0.4; reference probabilities are 0.5 and 0.5. At beta=1 the log-ratio difference is ln1.5=0.405465, logistic value is 0.6 and loss is −ln0.6=0.510826. If policy equals reference the difference is zero and loss is ln2 regardless of the reference preference. Swapping labels reverses the difference and raises this example’s loss to −ln0.4=0.916291.

Assumptions Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Both policy and reference follow the same update.

DPO avoids fitting and sampling from a separate reward model in this update, but remains dependent on the preference data and reference choice.

Separate training signals and algorithms, then inspect actual policy changes and proxy failures. Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.

Specifications and interfaces

Learning mode: executable DPO
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.
Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.

Interfaces

Inputs and state Input: Freeze a reference checkpoint at the declared branch point.
Result and check Output: Compute DPO from policy and frozen-reference response likelihoods with an explicit pair denominator.

Check your understanding

Which state receives gradients in the declared DPO branch?

Both policy and reference follow the same update.
Only the reference is trained.
The current policy; the branch reference remains fixed.

Answer and reasoning

The current policy; the branch reference remains fixed.

The current policy; the branch reference remains fixed. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.

Evidence and scope

Mode: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: executable DPO. Positive probabilities, fixed reference and consistent response token scoring. Core uses authored swap preferences and answer-plus-EOS scores, not collected human labels.