{
  "schemaVersion": 1,
  "id": "atlas-lesson-memory-working-set-placement",
  "title": "Place the working set, not just the model",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Assign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/memory-working-set-placement/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The placement framework is an engineering synthesis of established heterogeneous-memory and inference mechanisms. HBF placement remains a conditional scenario, not a claim that the cited runtimes support HBF.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-working-set-placement-3f85e16bc8008563.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#memory-working-set-placement",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "c8429b529b7a86cd13bb32e25d64ce85b28746d419a93e44f714143900fb17f8",
  "contentLength": 5637,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-working-set-placement.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-memory-working-set-placement.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Place the working set, not just the model",
      "start": 25,
      "end": 463
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 463,
      "end": 1724
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1724,
      "end": 1752
    },
    {
      "id": "section-4",
      "title": "Tier-capacity constraint",
      "start": 1752,
      "end": 2353
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2353,
      "end": 2637
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2637,
      "end": 3013
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3013,
      "end": 3270
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3270,
      "end": 3450
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3450,
      "end": 3479
    },
    {
      "id": "section-10",
      "title": "Why can a model fail to run even when its weight file fits in HBM?",
      "start": 3479,
      "end": 3909
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3909,
      "end": 4309
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4309,
      "end": 4860
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4860,
      "end": 4883
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4883,
      "end": 5120
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5120,
      "end": 5637
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#memory-working-set-placement"
    },
    {
      "title": "NVIDIA — Mastering LLM Techniques: Inference Optimization",
      "url": "https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/"
    },
    {
      "title": "ZeRO-Infinity: Breaking the GPU Memory Wall",
      "url": "https://arxiv.org/abs/2104.07857"
    },
    {
      "title": "Sandisk — release of first OCP HBF specification",
      "url": "https://investor.sandisk.com/node/8126/pdf"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/memory-working-set-placement-3f85e16bc8008563.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Full memory hierarchy and input path",
      "url": "https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/"
    },
    {
      "title": "Model graph and tensor intermediate representation",
      "url": "https://semiaifoundry.com/learn/atlas/graph-tensor-ir/"
    },
    {
      "title": "Training state across HBM, host memory and storage",
      "url": "https://semiaifoundry.com/learn/atlas/training-tier-placement/"
    },
    {
      "title": "Prefill, decode and the moving KV working set",
      "url": "https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/"
    },
    {
      "title": "MoE experts: resident capacity, routing and misses",
      "url": "https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/"
    },
    {
      "title": "Architecture is a contract between computation and silicon",
      "url": "https://semiaifoundry.com/learn/atlas/crg-architecture-contract/"
    }
  ],
  "recordHash": "fed6250a96f90aad3ac08e8b065c56d5b22bfc436f735baeec4999310985140b",
  "contentFormat": "text/markdown",
  "content": "Software · Atlas v1.3.0\n\n# Place the working set, not just the model\n\nAssign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#memory-working-set-placement)Register/SRAM ↔ HBM ↔ host ↔ proposed HBF ↔ storage; logical topology\n\nLearning objectiveAssign tensor state to tiers using lifetime, mutation, reuse distance, access size and latency requirements.\n\n## The mechanism\n\nA model is not one undifferentiated memory allocation. Parameters, activation tiles, optimizer state, KV cache, routing metadata and checkpoints have different lifetimes and traffic. Registers and SRAM serve small, frequently reused local data; HBM supports the accelerator’s active read/write state; host DDR or LPDDR offers another capacity and compute boundary. NAND-based HBF is an emerging candidate for read-heavy near-compute state, while local and remote storage retain datasets, checkpoints and less active material. These are roles, not a universal fixed ranking: effective placement depends on the actual links, controller behavior, software and workload.\n\n1.  Inventory each tensor/state object with its precision, dimensions, lifetime and mutation rate.\n2.  Identify the deadline for its next use, expected reuse distance and bytes consumed per fetch.\n3.  Reserve fast-tier space for concurrent workspaces and hot mutable state before allocating the remaining capacity to weights.\n4.  Define transfer, prefetch, eviction and consistency rules for each tier boundary, including persistent copies.\n5.  Profile misses, spills, writes and tail latency; move the object or change batching rather than judging only aggregate capacity.\n\n## Science and assumptions\n\n### Tier-capacity constraint\n\n$$\n\\sum_i x_{i,k}S_i+O_k\\leq C_k\n$$\n\nΣi xi,k Si + Ok ≤ Ck\n\nSymbols and units\n\nxi,k\n\nResident fraction/copy indicator of object i in tier k\n\ndimensionless\n\nSi\n\nObject size at its stored precision\n\nbytes\n\nOk\n\nWorkspace, staging and metadata overhead\n\nbytes\n\nCk\n\nUsable tier capacity\n\nbytes\n\nFor each tier k, resident objects plus reserved buffers and overhead must fit. x indicates the fraction or copy of object i resident there.\n\nAssumptions Use simultaneous live state, not sums of mutually exclusive tensors; copied buffers consume capacity in both tiers while alive.\n\n## Engineering example\n\nSynthetic 80 GiB usable tier: 45 GiB weights + 20 GiB KV + 12 GiB workspaces + 5 GiB other live state = 82 GiB. At least 2 GiB must be removed or reduced before allowing additional staging margin. A larger cold tier does not directly enlarge local workspace.\n\n## Distinctions and tradeoffs\n\nPlacement must account for all simultaneously live state at the relevant precision and execution phase.\n\nKeeping more weights local can displace KV or workspaces; offloading capacity creates transfer and synchronization work.\n\nMemory placement connects physical packaging choices to batch size, context, latency and productive accelerator use.\n\n## Specifications and interfaces\n\nPlacement record\n\nLifetime • reads/writes • reuse • granularity\n\nA parameter count alone is insufficient.\n\nTier boundary\n\nCapacity, link, service and consistency\n\nHost-accessible and device-local are different properties.\n\n### Interfaces\n\nTensor contract Input\n\nShape, precision, lifetime and required access pattern\n\nRuntime memory plan Output\n\nResidency, copies, transfer schedule and recovery state\n\n## Check your understanding\n\n### Why can a model fail to run even when its weight file fits in HBM?\n\n1.  Live KV, activations, workspaces and metadata also need capacity\n2.  Weights are the only memory allocation\n3.  Storage capacity fixes all device-memory limits\n\nAnswer and reasoning\n\nLive KV, activations, workspaces and metadata also need capacity\n\nPlacement must account for all simultaneously live state at the relevant precision and execution phase.\n\n## Evidence and scope\n\nThe placement framework is an engineering synthesis of established heterogeneous-memory and inference mechanisms. HBF placement remains a conditional scenario, not a claim that the cited runtimes support HBF.\n\nPlace read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.\n\n## Sources\n\n-   [NVIDIA — Mastering LLM Techniques: Inference Optimization](https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/)\n-   [ZeRO-Infinity: Breaking the GPU Memory Wall](https://arxiv.org/abs/2104.07857)\n-   [Sandisk — release of first OCP HBF specification](https://investor.sandisk.com/node/8126/pdf)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/memory-working-set-placement-3f85e16bc8008563.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Full memory hierarchy and input path](https://semiaifoundry.com/learn/atlas/memory-hierarchy-route/)\n-   [Model graph and tensor intermediate representation](https://semiaifoundry.com/learn/atlas/graph-tensor-ir/)\n\n### Related lessons\n\n-   [Training state across HBM, host memory and storage](https://semiaifoundry.com/learn/atlas/training-tier-placement/)\n-   [Prefill, decode and the moving KV working set](https://semiaifoundry.com/learn/atlas/prefill-decode-tier-placement/)\n-   [MoE experts: resident capacity, routing and misses](https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/)\n-   [Architecture is a contract between computation and silicon](https://semiaifoundry.com/learn/atlas/crg-architecture-contract/)"
}