{
  "schemaVersion": 1,
  "id": "atlas-lesson-moe",
  "title": "Mixture of experts",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Route each token to a subset of learned transformations.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/moe/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Transformer computation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This is the lesson’s equal-size top-k accounting example. Selected parameter count is not a complete FLOP, latency or quality model; shared experts, router work, imbalance, capacity limits and communication change actual execution.\n\nMode: worked example. Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/moe-5319c4e6acd569f7.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#moe",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "4f43ff034d1837b02dcdf04781421590d66783e28730a71d1a9e8c4de1e68944",
  "contentLength": 8897,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-moe.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-moe.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Mixture of experts",
      "start": 40,
      "end": 329
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 329,
      "end": 1805
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1805,
      "end": 1833
    },
    {
      "id": "section-4",
      "title": "Stored capacity versus activated computation",
      "start": 1833,
      "end": 3496
    },
    {
      "id": "section-5",
      "title": "Routing losses, sparse gradients and expert collapse · worked relation",
      "start": 3496,
      "end": 4633
    },
    {
      "id": "section-6",
      "title": "Engineering example",
      "start": 4633,
      "end": 5311
    },
    {
      "id": "section-7",
      "title": "Distinctions and tradeoffs",
      "start": 5311,
      "end": 5597
    },
    {
      "id": "section-8",
      "title": "Specifications and interfaces",
      "start": 5597,
      "end": 5800
    },
    {
      "id": "section-9",
      "title": "Interfaces",
      "start": 5800,
      "end": 5894
    },
    {
      "id": "section-10",
      "title": "Check your understanding",
      "start": 5894,
      "end": 5923
    },
    {
      "id": "section-11",
      "title": "Why can an MoE model have many stored but fewer active parameters?",
      "start": 5923,
      "end": 6241
    },
    {
      "id": "section-12",
      "title": "Does an auxiliary loss alone enforce the hard capacity of two?",
      "start": 6241,
      "end": 6797
    },
    {
      "id": "section-13",
      "title": "Historical context",
      "start": 6797,
      "end": 6820
    },
    {
      "id": "section-14",
      "title": "2021 · Switch Transformer simplifies sparse routing",
      "start": 6820,
      "end": 7140
    },
    {
      "id": "section-15",
      "title": "Evidence and scope",
      "start": 7140,
      "end": 7932
    },
    {
      "id": "section-16",
      "title": "Sources",
      "start": 7932,
      "end": 8265
    },
    {
      "id": "section-17",
      "title": "Continue exploring",
      "start": 8265,
      "end": 8288
    },
    {
      "id": "section-18",
      "title": "Foundations",
      "start": 8288,
      "end": 8386
    },
    {
      "id": "section-19",
      "title": "Related lessons",
      "start": 8386,
      "end": 8897
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#moe"
    },
    {
      "title": "Fedus et al. — Switch Transformers",
      "url": "https://arxiv.org/abs/2101.03961"
    },
    {
      "title": "Switch Transformers",
      "url": "https://arxiv.org/html/2101.03961v3"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/moe-5319c4e6acd569f7.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Feed-forward network",
      "url": "https://semiaifoundry.com/learn/atlas/feed-forward/"
    },
    {
      "title": "Data, tensor, pipeline, and state parallelism",
      "url": "https://semiaifoundry.com/learn/atlas/parallelism/"
    },
    {
      "title": "Collective communication library",
      "url": "https://semiaifoundry.com/learn/atlas/nccl/"
    },
    {
      "title": "Network interface adapter",
      "url": "https://semiaifoundry.com/learn/atlas/network-adapter/"
    },
    {
      "title": "A differentiable computation graph",
      "url": "https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/"
    },
    {
      "title": "Assemble an initialized causal language model",
      "url": "https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/"
    }
  ],
  "recordHash": "219967cf6856d2936c6ec084e0f3efaefae71d8f2b7ef897f4fb2f2a9aac4bb1",
  "contentFormat": "text/markdown",
  "content": "Transformer computation · Atlas v1.3.0\n\n# Mixture of experts\n\nRoute each token to a subset of learned transformations.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#moe)Logical structure · mapped onto physical hardware\n\nLearning objectiveSeparate total stored expert parameters from parameters selected for one token.\n\n## The mechanism\n\nA mixture-of-experts layer contains several expert networks and a router. The router selects a small subset for each token, then the system combines their outputs. Total stored parameters can therefore grow faster than the arithmetic used for one token. This creates an infrastructure problem: expert weights need capacity, routing can be imbalanced, and tokens may travel between devices. The experts are numerical functions, not independently conscious specialists.\n\nAn MoE router assigns tokens to selected experts, which changes both active computation and where gradients flow. A load-balancing auxiliary loss encourages use of experts under a particular formula; it is distinct from a hard capacity limit or dropping policy. The Switch top-one convention uses discrete dispatch fractions and mean router probabilities. Its dispatch counts are not differentiated through argmax. The lifecycle model is dense and does not execute expert routing or this auxiliary loss.\n\n1.  Score expert choices for each token.\n2.  Dispatch tokens to selected experts, possibly across devices.\n3.  Combine outputs and return them to the continuing representation.\n4.  Route tokens under an explicit top-k and tie convention.\n5.  Count dispatch fractions and mean router probabilities separately.\n6.  Evaluate the chosen auxiliary formula and identify its differentiable path.\n7.  Apply expert capacity and account for accepted, dropped or bypassed tokens independently.\n\n## Science and assumptions\n\n### Stored capacity versus activated computation\n\n$$\n\\begin{aligned}y&=\\sum_{i\\in S(x)}g_i(x)E_i(x)\\\\N_{\\mathrm{total}}&=N_{\\mathrm{shared}}+E N_{\\mathrm{expert}}\\\\N_{\\mathrm{active}}&\\approx N_{\\mathrm{shared}}+kN_{\\mathrm{expert}}\\end{aligned}\n$$\n\nThe output sums selected expert outputs weighted by the router. Stored parameters include all experts; approximate active parameters include only the selected k experts plus shared parameters.\n\nSymbols and units\n\nx\n\nInput token representation\n\nmodel-coordinate vector\n\ny\n\nCombined expert output\n\nmodel-coordinate vector\n\nS(x)\n\nSelected expert IDs for x\n\nset of k experts\n\ngᵢ(x)\n\nRouting weight for selected expert i\n\ndimensionless\n\nEᵢ(x)\n\nOutput of expert function i\n\nsame shape and units as y\n\nE\n\nTotal expert count (distinct from function Eᵢ)\n\ncount\n\nk\n\nExperts selected per token\n\ncount\n\nN\\_shared\n\nShared parameters included in this bookkeeping\n\nparameters\n\nN\\_expert\n\nParameters per equally sized expert\n\nparameters\n\nN\\_total\n\nTotal stored parameter count\n\nparameters\n\nN\\_active\n\nApproximate parameters activated for one token\n\nparameters\n\nA router selects a set S(x) containing k of E experts. The selected outputs receive routing weights $g_{i}$. With equally sized experts, increasing E can enlarge stored capacity without multiplying the number of expert matrices evaluated for each token. Communication and routing still consume real time.\n\nAssumptions This is a simplified top-k layer with equal expert sizes. Routing weights may be renormalized differently, capacity limits can drop or reroute tokens, and shared experts alter the count. Parameter activation is not a complete FLOP or latency model.\n\n### Routing losses, sparse gradients and expert collapse · worked relation\n\n$$\nL_{aux}=\\alpha N\\sum_i f_iP_i,\\quad f_i=\\frac1T\\sum_x\\mathbf1[\\arg\\max_j p_j(x)=i],\\quad P_i=\\frac1T\\sum_xp_i(x)\n$$\n\nCalculate routing loads and a declared auxiliary loss separately from expert capacity.\n\nSymbols and units\n\nL\\_aux\n\nSwitch-style auxiliary objective\n\ndimensionless\n\nalpha\n\nAuxiliary coefficient\n\ndimensionless\n\nN\n\nExperts\n\nexperts\n\nT\n\nRouted tokens\n\ntokens\n\nf\\_i\n\nDiscrete dispatch fraction\n\ndimensionless\n\nP\\_i\n\nMean router probability\n\ndimensionless\n\np\\_i(x)\n\nRouter probability for token x\n\ndimensionless\n\nSix tokens dispatch to three experts with counts \\[3,2,1\\], so f=\\[1/2,1/3,1/6\\]. Mean router probabilities are P=\\[1/2,1/3,1/6\\]. With alpha=0.01 and N=3, auxiliary loss is 0.01×3×(1/4+1/9+1/36)=0.011667. Uniform f=P=\\[1/3,1/3,1/3\\] gives 0.01. A hard capacity of two tokens per expert still rejects or bypasses one token from expert one under a declared policy.\n\nAssumptions Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing.\n\n## Engineering example\n\nSynthetic layer has eight experts with 1 million parameters each, plus 2 million shared parameters, and selects two experts per token. Stored parameters total 10 million; the simplified selected count is 4 million. At two bytes per stored parameter, weights occupy 20 MB before metadata.\n\nSix tokens dispatch to three experts with counts \\[3,2,1\\], so f=\\[1/2,1/3,1/6\\]. Mean router probabilities are P=\\[1/2,1/3,1/6\\]. With alpha=0.01 and N=3, auxiliary loss is 0.01×3×(1/4+1/9+1/36)=0.011667. Uniform f=P=\\[1/3,1/3,1/3\\] gives 0.01. A hard capacity of two tokens per expert still rejects or bypasses one token from expert one under a declared policy.\n\n## Distinctions and tradeoffs\n\nAn expert is a learned computation, not a separate humanlike mind.\n\nSparse activation saves arithmetic relative to activating every expert, but increases routing and load-balancing complexity.\n\nModel design and network topology become tightly connected.\n\n## Specifications and interfaces\n\nActive vs total\n\nDifferent parameter counts\n\nOnly selected experts execute per token.\n\nCommunication\n\nOften all-to-all\n\nDepends on expert placement and implementation.\n\n### Interfaces\n\nRouter Input\n\nToken vectors\n\nResidual stream Output\n\nWeighted expert outputs\n\n## Check your understanding\n\n### Why can an MoE model have many stored but fewer active parameters?\n\n1.  Only selected experts run for each token\n2.  Its memory is imaginary\n3.  All expert weights disappear after training\n\nAnswer and reasoning\n\nOnly selected experts run for each token\n\nRouting selects a subset of the available expert networks.\n\n### Does an auxiliary loss alone enforce the hard capacity of two?\n\n1.  Only if all model parameters are dense.\n2.  No; dispatch capacity and overflow behavior need a separate rule.\n3.  Yes; auxiliary losses physically prevent every overflow.\n\nAnswer and reasoning\n\nNo; dispatch capacity and overflow behavior need a separate rule.\n\nNo; dispatch capacity and overflow behavior need a separate rule. Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing.\n\n## Historical context\n\n### 2021 · Switch Transformer simplifies sparse routing\n\nThe Switch Transformer preprint investigated top-one expert routing and training techniques that made very large sparse Transformer models easier to train; it was published in JMLR in 2022.\n\n[Fedus et al. — Switch Transformers](https://arxiv.org/abs/2101.03961)\n\n## Evidence and scope\n\nThis is the lesson’s equal-size top-k accounting example. Selected parameter count is not a complete FLOP, latency or quality model; shared experts, router work, imbalance, capacity limits and communication change actual execution.\n\nMode: worked example. Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [Fedus et al. — Switch Transformers](https://arxiv.org/abs/2101.03961)\n-   [Switch Transformers](https://arxiv.org/html/2101.03961v3)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/moe-5319c4e6acd569f7.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Feed-forward network](https://semiaifoundry.com/learn/atlas/feed-forward/)\n\n### Related lessons\n\n-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)\n-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)\n-   [Network interface adapter](https://semiaifoundry.com/learn/atlas/network-adapter/)\n-   [A differentiable computation graph](https://semiaifoundry.com/learn/atlas/reverse-mode-autodiff/)\n-   [Assemble an initialized causal language model](https://semiaifoundry.com/learn/atlas/tiny-causal-decoder/)"
}