{"id":"moe","level":"model","name":"Mixture of experts","shortName":"Mixture of experts","role":"Route each token to a subset of learned transformations.","description":"A mixture-of-experts layer contains several expert networks and a router. The router selects a small subset for each token, then the system combines their outputs. Total stored parameters can therefore grow faster than the arithmetic used for one token. This creates an infrastructure problem: expert weights need capacity, routing can be imbalanced, and tokens may travel between devices. The experts are numerical functions, not independently conscious specialists.\n\nAn MoE router assigns tokens to selected experts, which changes both active computation and where gradients flow. A load-balancing auxiliary loss encourages use of experts under a particular formula; it is distinct from a hard capacity limit or dropping policy. The Switch top-one convention uses discrete dispatch fractions and mean router probabilities. Its dispatch counts are not differentiated through argmax. The lifecycle model is dense and does not execute expert routing or this auxiliary loss.","mechanism":["Score expert choices for each token.","Dispatch tokens to selected experts, possibly across devices.","Combine outputs and return them to the continuing representation.","Route tokens under an explicit top-k and tie convention.","Count dispatch fractions and mean router probabilities separately.","Evaluate the chosen auxiliary formula and identify its differentiable path.","Apply expert capacity and account for accepted, dropped or bypassed tokens independently."],"specs":[{"label":"Active vs total","value":"Different parameter counts","context":"Only selected experts execute per token."},{"label":"Communication","value":"Often all-to-all","context":"Depends on expert placement and implementation."}],"interfaces":[{"name":"Router","direction":"in","signal":"Token vectors"},{"name":"Residual stream","direction":"out","signal":"Weighted expert outputs"}],"tradeoff":"Sparse activation saves arithmetic relative to activating every expert, but increases routing and load-balancing complexity.","whyAI":"Model design and network topology become tightly connected.","related":["feed-forward","parallelism","nccl","network-adapter","reverse-mode-autodiff","tiny-causal-decoder"],"sources":[{"title":"Fedus et al. — Switch Transformers","url":"https://arxiv.org/abs/2101.03961"},{"title":"Switch Transformers","url":"https://arxiv.org/html/2101.03961v3"}],"scale":"Logical structure · mapped onto physical hardware","kind":"moe","misconception":"An expert is a learned computation, not a separate humanlike mind.","check":{"question":"Why can an MoE model have many stored but fewer active parameters?","options":["Only selected experts run for each token","Its memory is imaginary","All expert weights disappear after training"],"answer":0,"explanation":"Routing selects a subset of the available expert networks."},"branch":"Transformer computation","subbranch":"Conditional expert routing","science":[{"title":"Stored capacity versus activated computation","equation":"y = Σ[i∈S(x)] g_i(x)E_i(x); N_total = N_shared + E·N_expert; N_active ≈ N_shared + k·N_expert","explanation":"A router selects a set S(x) containing k of E experts. The selected outputs receive routing weights g_i. With equally sized experts, increasing E can enlarge stored capacity without multiplying the number of expert matrices evaluated for each token. Communication and routing still consume real time.","assumptions":"This is a simplified top-k layer with equal expert sizes. Routing weights may be renormalized differently, capacity limits can drop or reroute tokens, and shared experts alter the count. Parameter activation is not a complete FLOP or latency model."},{"title":"Routing losses, sparse gradients and expert collapse · worked relation","equation":"L_{aux}=\\alpha N\\sum_i f_iP_i,\\quad f_i=\\frac1T\\sum_x\\mathbf1[\\arg\\max_j p_j(x)=i],\\quad P_i=\\frac1T\\sum_xp_i(x)","explanation":"Six tokens dispatch to three experts with counts [3,2,1], so f=[1/2,1/3,1/6]. Mean router probabilities are P=[1/2,1/3,1/6]. With alpha=0.01 and N=3, auxiliary loss is 0.01×3×(1/4+1/9+1/36)=0.011667. Uniform f=P=[1/3,1/3,1/3] gives 0.01. A hard capacity of two tokens per expert still rejects or bypasses one token from expert one under a declared policy.","assumptions":"Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing."}],"history":[{"year":2021,"title":"Switch Transformer simplifies sparse routing","significance":"The Switch Transformer preprint investigated top-one expert routing and training techniques that made very large sparse Transformer models easier to train; it was published in JMLR in 2022.","source":{"title":"Fedus et al. — Switch Transformers","url":"https://arxiv.org/abs/2101.03961"}}],"prerequisites":["feed-forward"],"learningObjective":"Separate total stored expert parameters from parameters selected for one token.","engineeringExample":"Synthetic layer has eight experts with 1 million parameters each, plus 2 million shared parameters, and selects two experts per token. Stored parameters total 10 million; the simplified selected count is 4 million. At two bytes per stored parameter, weights occupy 20 MB before metadata.\n\nSix tokens dispatch to three experts with counts [3,2,1], so f=[1/2,1/3,1/6]. Mean router probabilities are P=[1/2,1/3,1/6]. With alpha=0.01 and N=3, auxiliary loss is 0.01×3×(1/4+1/9+1/36)=0.011667. Uniform f=P=[1/3,1/3,1/3] gives 0.01. A hard capacity of two tokens per expert still rejects or bypasses one token from expert one under a declared policy.","evidenceNotes":"This is the lesson’s equal-size top-k accounting example. Selected parameter count is not a complete FLOP, latency or quality model; shared experts, router work, imbalance, capacity limits and communication change actual execution.\n\nMode: worked example. Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","checks":[{"question":"Does an auxiliary loss alone enforce the hard capacity of two?","options":["Only if all model parameters are dense.","No; dispatch capacity and overflow behavior need a separate rule.","Yes; auxiliary losses physically prevent every overflow."],"answer":1,"explanation":"No; dispatch capacity and overflow behavior need a separate rule. Switch top-one auxiliary convention, before capacity handling. Discrete f is treated as nondifferentiable; P supplies a gradient path. Worked-only routing."}],"labId":"library","scope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.","investigationId":"execution","labScope":"Use the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism."}