{"id":"imagined-rollouts-policy-learning","name":"Learn behavior from a model and test in the environment","shortName":"Learn behavior from a model and test in the environment","role":"Compare a policy’s imagined return with return in the actual environment and identify model exploitation.","description":"A learned world model can supply imagined transitions for training an actor and critic. The actor then chooses actions and the critic estimates future returns. Training with imagined trajectories is different from running an online search at every real decision. Dreamer’s actor can act from its learned state without a fresh tree search; model-predictive control uses a different selection procedure. Both depend on how well the learned dynamics and reward model represent the states the policy visits. Optimizing an imperfect model can select actions that look excellent only inside that model.","mechanism":["Start imagined trajectories from recorded states with a stated model identity.","Roll actions through predicted transitions and rewards, respecting termination.","Train or compare policies under a declared return objective.","Execute the selected policy in the separate environment and record the gap, including states where the model was unsupported."],"learningObjective":"Compare a policy’s imagined return with return in the actual environment and identify model exploitation.","engineeringExample":"For two future rewards [1,2] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome.","evidenceNotes":"Mode: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compare a policy’s imagined return with return in the actual environment and identify model exploitation.","context":"Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Start imagined trajectories from recorded states with a stated model identity."},{"name":"Result and check","direction":"out","signal":"Compare a policy’s imagined return with return in the actual environment and identify model exploitation."}],"tradeoff":"Imagined experience can reduce real interactions but inherits model error. Shorter imagination reduces some error accumulation while limiting long-horizon credit assignment.","whyAI":"Distinguish predicting observations, choosing actions and controlling a changing environment. Compare a policy’s imagined return with return in the actual environment and identify model exploitation.","misconception":"It is a mistake to conclude: Imagined and actual returns are identical by definition.","check":{"question":"Why does the unsupported action look best during imagined optimization?","options":["Discounting guarantees the model cannot be exploited.","Its predicted return is overestimated relative to the separate environment.","Imagined and actual returns are identical by definition."],"answer":1,"explanation":"Its predicted return is overestimated relative to the separate environment. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards."},"prerequisites":["action-conditioned-world-model","policy-gradient-kl-control"],"related":["action-conditioned-world-model","policy-gradient-kl-control","vision-language-action-interface","diffusion-action-horizon"],"sources":[{"title":"Mastering Diverse Domains through World Models","url":"https://arxiv.org/html/2301.04104v2"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-world-action","subbranch":"Learn behavior from a model and test in the environment","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.","science":[{"title":"Learn behavior from a model and test in the environment · worked relation","equation":"G_t=\\sum_{k=0}^{H-1}\\gamma^k r_{t+k},\\qquad \\Delta_{model}=\\hat G-G_{environment}","explanation":"For two future rewards [1,2] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome.","assumptions":"Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards."}],"history":[]}