# Learn behavior from a model and test in the environment Canonical: https://semiaifoundry.com/learn/atlas/imagined-rollouts-policy-learning/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-world-action · Atlas v1.3.0 # Learn behavior from a model and test in the environment Compare a policy’s imagined return with return in the actual environment and identify model exploitation. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#imagined-rollouts-policy-learning)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCompare a policy’s imagined return with return in the actual environment and identify model exploitation. ## The mechanism A learned world model can supply imagined transitions for training an actor and critic. The actor then chooses actions and the critic estimates future returns. Training with imagined trajectories is different from running an online search at every real decision. Dreamer’s actor can act from its learned state without a fresh tree search; model-predictive control uses a different selection procedure. Both depend on how well the learned dynamics and reward model represent the states the policy visits. Optimizing an imperfect model can select actions that look excellent only inside that model. 1. Start imagined trajectories from recorded states with a stated model identity. 2. Roll actions through predicted transitions and rewards, respecting termination. 3. Train or compare policies under a declared return objective. 4. Execute the selected policy in the separate environment and record the gap, including states where the model was unsupported. ## Science and assumptions ### Learn behavior from a model and test in the environment · worked relation $$ G_t=\sum_{k=0}^{H-1}\gamma^k r_{t+k},\qquad \Delta_{model}=\hat G-G_{environment} $$ Compare a policy’s imagined return with return in the actual environment and identify model exploitation. Symbols and units G\_t Finite-horizon discounted return reward units gamma Discount factor dimensionless H Reward horizon steps r Reward at the stated transition boundary reward units hat G Return predicted by the model reward units Delta\_model Predicted minus environment return reward units For two future rewards \[1,2\] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome. Assumptions Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. ## Engineering example For two future rewards \[1,2\] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome. ## Distinctions and tradeoffs It is a mistake to conclude: Imagined and actual returns are identical by definition. Imagined experience can reduce real interactions but inherits model error. Shorter imagination reduces some error accumulation while limiting long-horizon credit assignment. Distinguish predicting observations, choosing actions and controlling a changing environment. Compare a policy’s imagined return with return in the actual environment and identify model exploitation. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Compare a policy’s imagined return with return in the actual environment and identify model exploitation. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. ### Interfaces Inputs and state Input Start imagined trajectories from recorded states with a stated model identity. Result and check Output Compare a policy’s imagined return with return in the actual environment and identify model exploitation. ## Check your understanding ### Why does the unsupported action look best during imagined optimization? 1. Discounting guarantees the model cannot be exploited. 2. Its predicted return is overestimated relative to the separate environment. 3. Imagined and actual returns are identical by definition. Answer and reasoning Its predicted return is overestimated relative to the separate environment. Its predicted return is overestimated relative to the separate environment. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. ## Evidence and scope Mode: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. ## Sources - [Mastering Diverse Domains through World Models](https://arxiv.org/html/2301.04104v2) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/imagined-rollouts-policy-learning-1c1807caf45ac53a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Predict transitions conditional on actions](https://semiaifoundry.com/learn/atlas/action-conditioned-world-model/) - [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/) ### Related lessons - [Translate learned representations into robot action units](https://semiaifoundry.com/learn/atlas/vision-language-action-interface/) - [Predict action chunks and replan with feedback](https://semiaifoundry.com/learn/atlas/diffusion-action-horizon/)