{
  "schemaVersion": 1,
  "id": "atlas-lesson-expert-routing-budget",
  "title": "MoE routing, active parameters and resident state",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Balances conditional expert computation against placement, communication and capacity.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/expert-routing-budget/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Model–machine co-design",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Routing policy and storage placement are model-specific. The toy traffic estimate does not describe any named production model or guarantee expert balance.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/expert-routing-budget-274a075fef73b09f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#expert-routing-budget",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "992f6380ef8f4137b5b196ce2481e454d4c582b1609e941305665f21626beaed",
  "contentLength": 5190,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-expert-routing-budget.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-expert-routing-budget.md",
  "sections": [
    {
      "id": "section-1",
      "title": "MoE routing, active parameters and resident state",
      "start": 40,
      "end": 408
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 408,
      "end": 1675
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1675,
      "end": 1703
    },
    {
      "id": "section-4",
      "title": "Simple dispatch volume",
      "start": 1703,
      "end": 2523
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2523,
      "end": 2747
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2747,
      "end": 3243
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3243,
      "end": 3515
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3515,
      "end": 3687
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3687,
      "end": 3716
    },
    {
      "id": "section-10",
      "title": "Why can an MoE model require large memory despite low active parameters per token?",
      "start": 3716,
      "end": 4189
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4189,
      "end": 4513
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4513,
      "end": 4846
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4846,
      "end": 4869
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4869,
      "end": 5060
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5060,
      "end": 5190
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#expert-routing-budget"
    },
    {
      "title": "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer",
      "url": "https://arxiv.org/abs/1701.06538"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/expert-routing-budget-274a075fef73b09f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Mixture of experts",
      "url": "https://semiaifoundry.com/learn/atlas/moe/"
    },
    {
      "title": "Data, tensor, pipeline, and state parallelism",
      "url": "https://semiaifoundry.com/learn/atlas/parallelism/"
    },
    {
      "title": "Autoregressive decode and weight traffic",
      "url": "https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/"
    }
  ],
  "recordHash": "89318ee4382191ce0ec6792c3b2c6845af863d50242479293ba92294ce01eae3",
  "contentFormat": "text/markdown",
  "content": "Model–machine co-design · Atlas v1.3.0\n\n# MoE routing, active parameters and resident state\n\nBalances conditional expert computation against placement, communication and capacity.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#expert-routing-budget)Accelerator group to serving pool; routes schematic\n\nLearning objectiveSeparate MoE’s active computation savings from total state and routing costs.\n\n## The mechanism\n\nA mixture-of-experts layer sends each token to a selected subset of expert networks. This can increase total model parameters without activating all of them for every token. The inactive experts still need storage somewhere, and tokens must reach the devices holding their selected experts. Routing imbalance creates overloaded experts, padding or dropped work depending on the design. Small per-expert batches may also use matrix units inefficiently. The sparsely gated MoE research established conditional computation and load-balancing concerns. A useful resource card therefore records total parameters, active parameters, routing policy and communication rather than replacing model size with one active-parameter number.\n\n1.  Calculate router scores and select the permitted experts for each token under the model’s policy.\n2.  Group tokens by destination expert and check expert capacity or load-balancing constraints.\n3.  Dispatch activations to the devices holding those experts, including remote transfers where necessary.\n4.  Execute the resulting expert batches and account for padding, stragglers or sparse utilization.\n5.  Return and combine outputs, then measure total state, active arithmetic and communication as separate budgets.\n\n## Science and assumptions\n\n### Simple dispatch volume\n\n$$\nQ_{\\mathrm{dispatch}}=T k d s f_{\\mathrm{remote}}\n$$\n\nRemote dispatch bytes equal tokens times experts per token times activation width times bytes per element times the remote-dispatch fraction.\n\nSymbols and units\n\nQ\\_dispatch\n\nOne-way remote activation payload\n\nbytes\n\nT\n\nTokens dispatched\n\ntokens\n\nk\n\nSelected experts per token\n\nexperts/token\n\nd\n\nActivation width copied per dispatch\n\nelements/dispatch\n\ns\n\nStored activation element size\n\nbytes/element\n\nf\\_remote\n\nFraction of expert dispatches crossing the selected boundary\n\nfraction\n\nT is tokens, k selected experts/token, d activation elements, s bytes/element and fremote the remote fraction. Q is bytes.\n\nAssumptions Counts one-way activation payload only; return data, metadata, duplicates and protocol overhead are additional.\n\n## Engineering example\n\nSynthetic dispatch: 1,000 tokens, two experts per token, 4,096-element activations, two bytes/element and half remote gives 8,192,000 remote payload bytes one way, before return outputs or metadata.\n\n## Distinctions and tradeoffs\n\nInactive parameters do not vanish from memory, and fewer active FLOPS do not guarantee proportionally lower token latency.\n\nSparse activation saves arithmetic relative to executing every expert, but adds routing, load imbalance, fragmented batches and potentially large remote memory traffic.\n\nMoE can shift a bottleneck from dense arithmetic toward expert storage and all-to-all communication without reducing all system costs in proportion to active parameters.\n\n## Specifications and interfaces\n\nTwo parameter counts\n\nResident total and active per token\n\nShared layers and routing overhead must also be included.\n\nRouting traffic\n\nTokens × selected experts × activation bytes\n\nRemote fraction and return traffic depend on placement.\n\n### Interfaces\n\nToken activations and router Input\n\nExpert choices, weights and capacity decisions\n\nExpert outputs Output\n\nDispatched, computed and combined token results\n\n## Check your understanding\n\n### Why can an MoE model require large memory despite low active parameters per token?\n\n1.  All experts must remain stored or be fetched when selected\n2.  The router deletes inactive weights permanently\n3.  Active FLOPS and memory bytes are identical quantities\n\nAnswer and reasoning\n\nAll experts must remain stored or be fetched when selected\n\nConditional execution changes which expert weights participate in a token’s work; it does not eliminate the total model state.\n\n## Evidence and scope\n\nRouting policy and storage placement are model-specific. The toy traffic estimate does not describe any named production model or guarantee expert balance.\n\nNo current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.\n\n## Sources\n\n-   [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/expert-routing-budget-274a075fef73b09f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Mixture of experts](https://semiaifoundry.com/learn/atlas/moe/)\n-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)\n\n### Related lessons\n\n-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)"
}