{
  "schemaVersion": 1,
  "id": "atlas-lesson-training-tier-placement",
  "title": "Training state across HBM, host memory and storage",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Distinguish mutable training state, recomputation, offload and durable checkpoints.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/training-tier-placement/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "DeepSpeed documents optimizer offload to the host CPU; ZeRO-Infinity studies GPU/CPU/NVMe state management. Neither source establishes an HBF training implementation or its write endurance.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/training-tier-placement-fa59deb9ce7f62c5.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#training-tier-placement",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "b630d0777d8617493042a8aabb7021398cc2d7a7ed5f684cb396d471c7c9179f",
  "contentLength": 5450,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-training-tier-placement.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-training-tier-placement.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Training state across HBM, host memory and storage",
      "start": 25,
      "end": 418
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 418,
      "end": 1580
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1580,
      "end": 1608
    },
    {
      "id": "section-4",
      "title": "State inventory before sharding",
      "start": 1608,
      "end": 2441
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2441,
      "end": 2705
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2705,
      "end": 3148
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3148,
      "end": 3435
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3435,
      "end": 3622
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3622,
      "end": 3651
    },
    {
      "id": "section-10",
      "title": "Which state is inherently write-active during ordinary optimizer updates?",
      "start": 3651,
      "end": 4033
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4033,
      "end": 4399
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4399,
      "end": 4788
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4788,
      "end": 4811
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4811,
      "end": 5062
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5062,
      "end": 5450
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#training-tier-placement"
    },
    {
      "title": "DeepSpeed — ZeRO-Offload tutorial",
      "url": "https://www.deepspeed.ai/tutorials/zero-offload/"
    },
    {
      "title": "ZeRO-Infinity: Breaking the GPU Memory Wall",
      "url": "https://arxiv.org/abs/2104.07857"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/training-tier-placement-fa59deb9ce7f62c5.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training state and activation rematerialization",
      "url": "https://semiaifoundry.com/learn/atlas/training-memory-state/"
    },
    {
      "title": "Place the working set, not just the model",
      "url": "https://semiaifoundry.com/learn/atlas/memory-working-set-placement/"
    },
    {
      "title": "Checkpointing, failure and retained useful work",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/"
    },
    {
      "title": "Parallelism as a memory and communication budget",
      "url": "https://semiaifoundry.com/learn/atlas/parallelism-communication-budget/"
    },
    {
      "title": "HBF: a read-oriented tier and its evidence boundary",
      "url": "https://semiaifoundry.com/learn/atlas/hbf-read-tier-contract/"
    }
  ],
  "recordHash": "6ae9e76e056a6d0067eac15b97ea2779be93bbe98d1510245e712f644415c3ee",
  "contentFormat": "text/markdown",
  "content": "Software · Atlas v1.3.0\n\n# Training state across HBM, host memory and storage\n\nDistinguish mutable training state, recomputation, offload and durable checkpoints.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#training-tier-placement)Tensor state over a training step and across memory tiers; abstraction\n\nLearning objectiveDistinguish mutable training state, recomputation, offload and durable checkpoints.\n\n## The mechanism\n\nTraining retains values for the forward pass, computes gradients and updates parameters and optimizer state. The fastest tier must therefore accommodate more than inference weights. Activation checkpointing stores selected intermediates and recomputes others; persistent checkpoints instead save restartable state outside the live execution path. State sharding changes ownership across devices, while CPU/NVMe offload moves both capacity and work across interfaces. A read-oriented flash tier cannot simply inherit the role of frequently updated optimizer or gradient storage: its write service, software support and movement schedule must be demonstrated.\n\n1.  Separate parameters, gradients, master weights, optimizer moments, live activations and temporary buffers.\n2.  Set precision and sharding rules independently for each class of state.\n3.  Choose where optimizer computation occurs, keeping its mutable state and transfer requirements consistent.\n4.  Model recomputation and offload as extra work or movement rather than free capacity.\n5.  Save version-consistent durable checkpoints and measure recovery alongside steady step time.\n\n## Science and assumptions\n\n### State inventory before sharding\n\n$$\nM_{\\mathrm{state}}=P(b_w+b_g+b_m+b_o)+A+O\n$$\n\nMstate = P (bw + bg + bm + bo) + A + O\n\nSymbols and units\n\nMstate\n\nLive model/training memory estimate\n\nbytes\n\nP\n\nParameter count\n\nparameters\n\nbw\n\nStored weight bytes per parameter\n\nbytes/parameter\n\nbg\n\nGradient bytes per parameter\n\nbytes/parameter\n\nbm\n\nSeparate master-weight bytes per parameter\n\nbytes/parameter\n\nbo\n\nOptimizer-state bytes per parameter\n\nbytes/parameter\n\nA\n\nRetained live activations\n\nbytes\n\nO\n\nOther live workspace and metadata\n\nbytes\n\nPer-parameter byte terms represent the explicitly chosen training representation. bo includes all optimizer-state values per parameter.\n\nAssumptions Dense unsharded inventory; omit absent state such as a separate master copy, then apply actual partition/replication factors by tensor class.\n\n## Engineering example\n\nSynthetic mixed-precision Adam inventory: 1 billion parameters × (2 weight + 2 gradient + 4 master + 8 moment bytes) = 16 GB before activations, workspace, sharding or replication. It is one representation, not a rule for every optimizer.\n\n## Distinctions and tradeoffs\n\nTraining repeatedly updates gradient/optimizer/parameter state, so read bandwidth alone cannot characterize an appropriate tier.\n\nOffload and recomputation enable larger models at the cost of movement or extra arithmetic; the best choice depends on overlap and recovery requirements.\n\nUseful training progress depends on completed and recoverable updates, not the largest parameter count that can be allocated.\n\n## Specifications and interfaces\n\nTwo checkpoint meanings\n\nActivation recomputation versus durable restart\n\nSaving an activation is not saving a recoverable training run.\n\nOffload boundary\n\nData and computation placement\n\nA CPU optimizer also consumes CPU compute and memory bandwidth.\n\n### Interfaces\n\nForward/backward tensors Input\n\nActivations, gradients and parameter version\n\nUpdate and durable state Output\n\nNew parameters, optimizer state and recoverable checkpoint\n\n## Check your understanding\n\n### Which state is inherently write-active during ordinary optimizer updates?\n\n1.  Gradients and optimizer state\n2.  Only the immutable tokenizer vocabulary\n3.  Only a finalized archival checkpoint\n\nAnswer and reasoning\n\nGradients and optimizer state\n\nTraining repeatedly updates gradient/optimizer/parameter state, so read bandwidth alone cannot characterize an appropriate tier.\n\n## Evidence and scope\n\nDeepSpeed documents optimizer offload to the host CPU; ZeRO-Infinity studies GPU/CPU/NVMe state management. Neither source establishes an HBF training implementation or its write endurance.\n\nEnable the optimizer-state option to budget weights and master/moment state; gradients, activations, workspaces and actual tier transfers are omitted.\n\n## Sources\n\n-   [DeepSpeed — ZeRO-Offload tutorial](https://www.deepspeed.ai/tutorials/zero-offload/)\n-   [ZeRO-Infinity: Breaking the GPU Memory Wall](https://arxiv.org/abs/2104.07857)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/training-tier-placement-fa59deb9ce7f62c5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Training state and activation rematerialization](https://semiaifoundry.com/learn/atlas/training-memory-state/)\n-   [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/)\n\n### Related lessons\n\n-   [Checkpointing, failure and retained useful work](https://semiaifoundry.com/learn/atlas/checkpoint-retained-work/)\n-   [Parallelism as a memory and communication budget](https://semiaifoundry.com/learn/atlas/parallelism-communication-budget/)\n-   [HBF: a read-oriented tier and its evidence boundary](https://semiaifoundry.com/learn/atlas/hbf-read-tier-contract/)"
}