{
  "schemaVersion": 1,
  "id": "atlas-lesson-moe-expert-tier-placement",
  "title": "MoE experts: resident capacity, routing and misses",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Explain why sparse activation reduces arithmetic without eliminating expert storage or transfer constraints.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "DeepSpeed supports the distinction between expert distribution and communication scheduling. HBF-based expert caching here is an explicit architectural scenario, not a feature claimed for that runtime.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/moe-expert-tier-placement-d2d76a649e40c474.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#moe-expert-tier-placement",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "6b5bf3b4de32dcf4495444e142fee973f963cabcfdf7638d79cfb9a7ecec17f3",
  "contentLength": 5283,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-moe-expert-tier-placement.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-moe-expert-tier-placement.md",
  "sections": [
    {
      "id": "section-1",
      "title": "MoE experts: resident capacity, routing and misses",
      "start": 25,
      "end": 473
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 473,
      "end": 1581
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1581,
      "end": 1609
    },
    {
      "id": "section-4",
      "title": "Expert-weight miss traffic",
      "start": 1609,
      "end": 2217
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2217,
      "end": 2498
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2498,
      "end": 2913
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2913,
      "end": 3153
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3153,
      "end": 3314
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3314,
      "end": 3343
    },
    {
      "id": "section-10",
      "title": "Each token selects two experts. Must a batch need only two experts resident?",
      "start": 3343,
      "end": 3725
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3725,
      "end": 4118
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4118,
      "end": 4532
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4532,
      "end": 4555
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4555,
      "end": 4922
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4922,
      "end": 5283
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#moe-expert-tier-placement"
    },
    {
      "title": "DeepSpeed — MoE inference",
      "url": "https://www.deepspeed.ai/tutorials/mixture-of-experts-inference/"
    },
    {
      "title": "Sandisk — release of first OCP HBF specification",
      "url": "https://investor.sandisk.com/node/8126/pdf"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/moe-expert-tier-placement-d2d76a649e40c474.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "MoE routing, active parameters and resident state",
      "url": "https://semiaifoundry.com/learn/atlas/expert-routing-budget/"
    },
    {
      "title": "Place the working set, not just the model",
      "url": "https://semiaifoundry.com/learn/atlas/memory-working-set-placement/"
    },
    {
      "title": "HBF latency, transfer granularity and prefetch",
      "url": "https://semiaifoundry.com/learn/atlas/hbf-latency-prefetch/"
    },
    {
      "title": "Collective traffic and congestion",
      "url": "https://semiaifoundry.com/learn/atlas/fabric-collective-traffic/"
    },
    {
      "title": "From memory placement to network, power and cooling",
      "url": "https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/"
    },
    {
      "title": "Communication is the realization medium",
      "url": "https://semiaifoundry.com/learn/atlas/crg-communication-graph/"
    }
  ],
  "recordHash": "d975dcb93f30f15f49f5f548968cc33f31d46d4ac7084fa9aaf9f7bb7e1745f7",
  "contentFormat": "text/markdown",
  "content": "Software · Atlas v1.3.0\n\n# MoE experts: resident capacity, routing and misses\n\nExplain why sparse activation reduces arithmetic without eliminating expert storage or transfer constraints.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#moe-expert-tier-placement)Router decisions, expert residency and physical link traffic; abstraction\n\nLearning objectiveExplain why sparse activation reduces arithmetic without eliminating expert storage or transfer constraints.\n\n## The mechanism\n\nA mixture-of-experts model selects only some experts for each token, but the full parameter set still needs a valid storage and delivery strategy. A larger batch can touch many distinct experts even when each token chooses only a few. Frequently used experts may remain in fast memory; inactive expert weights can occupy another tier if the runtime can bring them in before use. Router outcomes depend on the current hidden state, so advance prediction can be imperfect. Expert parallelism also moves token activations across devices, which is a different traffic stream from fetching missing expert weights.\n\n1.  Separate shared parameters, the full expert inventory and per-token selected experts.\n2.  Measure how many distinct expert weights a batch touches and which remain cached.\n3.  Account for weight misses, staging capacity and latency; track prefetch precision and wasted bytes.\n4.  Route token activations to resident experts using the actual expert-parallel topology.\n5.  Evaluate load imbalance, missed deadlines and quality-preserving routing behavior alongside throughput.\n\n## Science and assumptions\n\n### Expert-weight miss traffic\n\n$$\nV_{\\mathrm{miss}}=\\sum_{e\\in\\mathcal{M}}P_e b_e\n$$\n\nVmiss = Σe∈M Pe be\n\nSymbols and units\n\nVmiss\n\nTransferred expert-weight bytes\n\nbytes\n\nM\n\nDistinct expert cache misses in the window\n\nset\n\nPe\n\nParameters in missing expert e\n\nparameters\n\nbe\n\nStored bytes per parameter\n\nbytes/parameter\n\nThe set M contains distinct missing experts required during the measured reuse window; count each transferred expert once in that window.\n\nAssumptions Whole expert transfer at specified stored precision, no shared/sub-tensor fetch or compression metadata; streaming changes the model.\n\n## Engineering example\n\nSynthetic example: four distinct missed experts of 2 GiB each require 8 GiB of weight transfer at the stated precision. Ten token assignments to the same cached expert do not require ten new copies of its weights, although token dispatch still moves data.\n\n## Distinctions and tradeoffs\n\nThe union of selected experts across a batch can be much larger than the per-token selection count.\n\nCaching popular experts saves movement but occupies scarce fast-tier space; inaccurate prefetch can consume bandwidth needed for actual misses.\n\nMoE capacity claims become meaningful only when routing, memory placement and interconnect scheduling can deliver selected work on time.\n\n## Specifications and interfaces\n\nTwo traffic streams\n\nExpert weights and token dispatch\n\nThey may use different paths and cannot be conflated.\n\nSparse activity\n\nFew experts per token\n\nA batch may still touch much of the expert inventory.\n\n### Interfaces\n\nRouter output Input\n\nSelected experts and token assignments\n\nExpert execution Output\n\nDelivered weights, routed activations and combined output\n\n## Check your understanding\n\n### Each token selects two experts. Must a batch need only two experts resident?\n\n1.  No: tokens can choose different experts\n2.  Yes: top-2 means only two experts exist\n3.  Yes: unused experts have no stored weights\n\nAnswer and reasoning\n\nNo: tokens can choose different experts\n\nThe union of selected experts across a batch can be much larger than the per-token selection count.\n\n## Evidence and scope\n\nDeepSpeed supports the distinction between expert distribution and communication scheduling. HBF-based expert caching here is an explicit architectural scenario, not a feature claimed for that runtime.\n\nPlace read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.\n\n## Sources\n\n-   [DeepSpeed — MoE inference](https://www.deepspeed.ai/tutorials/mixture-of-experts-inference/)\n-   [Sandisk — release of first OCP HBF specification](https://investor.sandisk.com/node/8126/pdf)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/moe-expert-tier-placement-d2d76a649e40c474.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [MoE routing, active parameters and resident state](https://semiaifoundry.com/learn/atlas/expert-routing-budget/)\n-   [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/)\n-   [HBF latency, transfer granularity and prefetch](https://semiaifoundry.com/learn/atlas/hbf-latency-prefetch/)\n\n### Related lessons\n\n-   [Collective traffic and congestion](https://semiaifoundry.com/learn/atlas/fabric-collective-traffic/)\n-   [From memory placement to network, power and cooling](https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/)\n-   [Communication is the realization medium](https://semiaifoundry.com/learn/atlas/crg-communication-graph/)"
}