{
  "schemaVersion": 1,
  "id": "atlas-lesson-imagined-rollouts-policy-learning",
  "title": "Learn behavior from a model and test in the environment",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compare a policy’s imagined return with return in the actual environment and identify model exploitation.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/imagined-rollouts-policy-learning/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-world-action",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/imagined-rollouts-policy-learning-1c1807caf45ac53a.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#imagined-rollouts-policy-learning",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ac6a02ecacc2081684157f630fa0b2bb83209a2043ac8b043e43cba5db511bb2",
  "contentLength": 6430,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-imagined-rollouts-policy-learning.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-imagined-rollouts-policy-learning.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Learn behavior from a model and test in the environment",
      "start": 45,
      "end": 510
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 510,
      "end": 1489
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1489,
      "end": 1517
    },
    {
      "id": "section-4",
      "title": "Learn behavior from a model and test in the environment · worked relation",
      "start": 1517,
      "end": 2664
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2664,
      "end": 3060
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3060,
      "end": 3554
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3554,
      "end": 4018
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4018,
      "end": 4270
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4270,
      "end": 4299
    },
    {
      "id": "section-10",
      "title": "Why does the unsupported action look best during imagined optimization?",
      "start": 4299,
      "end": 4926
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4926,
      "end": 5565
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5565,
      "end": 5880
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5880,
      "end": 5903
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5903,
      "end": 6155
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6155,
      "end": 6430
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#imagined-rollouts-policy-learning"
    },
    {
      "title": "Mastering Diverse Domains through World Models",
      "url": "https://arxiv.org/html/2301.04104v2"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/imagined-rollouts-policy-learning-1c1807caf45ac53a.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Predict transitions conditional on actions",
      "url": "https://semiaifoundry.com/learn/atlas/action-conditioned-world-model/"
    },
    {
      "title": "Rollouts, advantages and PPO constraints",
      "url": "https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/"
    },
    {
      "title": "Translate learned representations into robot action units",
      "url": "https://semiaifoundry.com/learn/atlas/vision-language-action-interface/"
    },
    {
      "title": "Predict action chunks and replan with feedback",
      "url": "https://semiaifoundry.com/learn/atlas/diffusion-action-horizon/"
    }
  ],
  "recordHash": "528d29d4fe077b16bec1ff6de897844d7924e3a32b17c3d416944b752f311247",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-world-action · Atlas v1.3.0\n\n# Learn behavior from a model and test in the environment\n\nCompare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#imagined-rollouts-policy-learning)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\n## The mechanism\n\nA learned world model can supply imagined transitions for training an actor and critic. The actor then chooses actions and the critic estimates future returns. Training with imagined trajectories is different from running an online search at every real decision. Dreamer’s actor can act from its learned state without a fresh tree search; model-predictive control uses a different selection procedure. Both depend on how well the learned dynamics and reward model represent the states the policy visits. Optimizing an imperfect model can select actions that look excellent only inside that model.\n\n1.  Start imagined trajectories from recorded states with a stated model identity.\n2.  Roll actions through predicted transitions and rewards, respecting termination.\n3.  Train or compare policies under a declared return objective.\n4.  Execute the selected policy in the separate environment and record the gap, including states where the model was unsupported.\n\n## Science and assumptions\n\n### Learn behavior from a model and test in the environment · worked relation\n\n$$\nG_t=\\sum_{k=0}^{H-1}\\gamma^k r_{t+k},\\qquad \\Delta_{model}=\\hat G-G_{environment}\n$$\n\nCompare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\nSymbols and units\n\nG\\_t\n\nFinite-horizon discounted return\n\nreward units\n\ngamma\n\nDiscount factor\n\ndimensionless\n\nH\n\nReward horizon\n\nsteps\n\nr\n\nReward at the stated transition boundary\n\nreward units\n\nhat G\n\nReturn predicted by the model\n\nreward units\n\nDelta\\_model\n\nPredicted minus environment return\n\nreward units\n\nFor two future rewards \\[1,2\\] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome.\n\nAssumptions Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.\n\n## Engineering example\n\nFor two future rewards \\[1,2\\] and discount 0.9, return is 1+0.9×2=2.8. In an authored choice, a safe action has imagined and actual return 2, while an unsupported action has imagined return 5 and actual return −1. Maximizing imagined return selects the latter, with a model-return gap of 6. A better score in the model therefore can accompany a worse real task outcome.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Imagined and actual returns are identical by definition.\n\nImagined experience can reduce real interactions but inherits model error. Shorter imagination reduces some error accumulation while limiting long-horizon credit assignment.\n\nDistinguish predicting observations, choosing actions and controlling a changing environment. Compare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\nAuthored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.\n\n### Interfaces\n\nInputs and state Input\n\nStart imagined trajectories from recorded states with a stated model identity.\n\nResult and check Output\n\nCompare a policy’s imagined return with return in the actual environment and identify model exploitation.\n\n## Check your understanding\n\n### Why does the unsupported action look best during imagined optimization?\n\n1.  Discounting guarantees the model cannot be exploited.\n2.  Its predicted return is overestimated relative to the separate environment.\n3.  Imagined and actual returns are identical by definition.\n\nAnswer and reasoning\n\nIts predicted return is overestimated relative to the separate environment.\n\nIts predicted return is overestimated relative to the separate environment. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.\n\n## Evidence and scope\n\nMode: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Authored deterministic rewards and finite horizon; no learned actor/critic or robot is executed here. Terminal states must stop or otherwise explicitly zero further rewards.\n\n## Sources\n\n-   [Mastering Diverse Domains through World Models](https://arxiv.org/html/2301.04104v2)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/imagined-rollouts-policy-learning-1c1807caf45ac53a.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Predict transitions conditional on actions](https://semiaifoundry.com/learn/atlas/action-conditioned-world-model/)\n-   [Rollouts, advantages and PPO constraints](https://semiaifoundry.com/learn/atlas/policy-gradient-kl-control/)\n\n### Related lessons\n\n-   [Translate learned representations into robot action units](https://semiaifoundry.com/learn/atlas/vision-language-action-interface/)\n-   [Predict action chunks and replan with feedback](https://semiaifoundry.com/learn/atlas/diffusion-action-horizon/)"
}