Learn behavior from a model and test in the environment

The mechanism

A learned world model can supply imagined transitions for training an actor and critic. The actor then chooses actions and the critic estimates future returns. Training with imagined trajectories is different from running an online search at every real decision. Dreamer’s actor can act from its learned state without a fresh tree search; model-predictive control uses a different selection procedure. Both depend on how well the learned dynamics and reward model represent the states the policy visits. Optimizing an imperfect model can select actions that look excellent only inside that model.

Start imagined trajectories from recorded states with a stated model identity.
Roll actions through predicted transitions and rewards, respecting termination.
Train or compare policies under a declared return objective.
Execute the selected policy in the separate environment and record the gap, including states where the model was unsupported.

Science and assumptions

Learn behavior from a model and test in the environment · worked relation

G_t=\sum_{k=0}^{H-1}\gamma^k r_{t+k},\qquad \Delta_{model}=\hat G-G_{environment}

Compare a policy’s imagined return with return in the actual environment and identify model exploitation.

Symbols and units

G_t: Finite-horizon discounted return
reward units
gamma: Discount factor
dimensionless
H: Reward horizon
steps
r: Reward at the stated transition boundary
reward units
hat G: Return predicted by the model
reward units
Delta_model: Predicted minus environment return
reward units

For two future rewards [1,2] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome.

Assumptions Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: Imagined and actual returns are identical by definition.

Imagined experience can reduce real interactions but inherits model error. Shorter imagination reduces some error accumulation while limiting long-horizon credit assignment.

Distinguish predicting observations, choosing actions and controlling a changing environment. Compare a policy’s imagined return with return in the actual environment and identify model exploitation.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Compare a policy’s imagined return with return in the actual environment and identify model exploitation.
Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.

Interfaces

Inputs and state Input: Start imagined trajectories from recorded states with a stated model identity.
Result and check Output: Compare a policy’s imagined return with return in the actual environment and identify model exploitation.

Check your understanding

Why does the unsupported action look best during imagined optimization?

Discounting guarantees the model cannot be exploited.
Its predicted return is overestimated relative to the separate environment.
Imagined and actual returns are identical by definition.

Answer and reasoning

Its predicted return is overestimated relative to the separate environment.

Its predicted return is overestimated relative to the separate environment. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.

Evidence and scope

Mode: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.