{"id":"expert-routing-budget","level":"software","name":"MoE routing, active parameters and resident state","shortName":"MoE routing, active parameters and resident state","role":"Balances conditional expert computation against placement, communication and capacity.","description":"A mixture-of-experts layer sends each token to a selected subset of expert networks. This can increase total model parameters without activating all of them for every token. The inactive experts still need storage somewhere, and tokens must reach the devices holding their selected experts. Routing imbalance creates overloaded experts, padding or dropped work depending on the design. Small per-expert batches may also use matrix units inefficiently. The sparsely gated MoE research established conditional computation and load-balancing concerns. A useful resource card therefore records total parameters, active parameters, routing policy and communication rather than replacing model size with one active-parameter number.","specs":[{"label":"Two parameter counts","value":"Resident total and active per token","context":"Shared layers and routing overhead must also be included."},{"label":"Routing traffic","value":"Tokens × selected experts × activation bytes","context":"Remote fraction and return traffic depend on placement."}],"interfaces":[{"name":"Token activations and router","direction":"in","signal":"Expert choices, weights and capacity decisions"},{"name":"Expert outputs","direction":"out","signal":"Dispatched, computed and combined token results"}],"tradeoff":"Sparse activation saves arithmetic relative to executing every expert, but adds routing, load imbalance, fragmented batches and potentially large remote memory traffic.","whyAI":"MoE can shift a bottleneck from dense arithmetic toward expert storage and all-to-all communication without reducing all system costs in proportion to active parameters.","related":["moe","parallelism","decode-bandwidth-trace"],"sources":[{"title":"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer","url":"https://arxiv.org/abs/1701.06538"}],"scale":"Accelerator group to serving pool; routes schematic","kind":"workload","branch":"Model–machine co-design","subbranch":"Distributed work and serving choices","mechanism":["Calculate router scores and select the permitted experts for each token under the model’s policy.","Group tokens by destination expert and check expert capacity or load-balancing constraints.","Dispatch activations to the devices holding those experts, including remote transfers where necessary.","Execute the resulting expert batches and account for padding, stragglers or sparse utilization.","Return and combine outputs, then measure total state, active arithmetic and communication as separate budgets."],"misconception":"Inactive parameters do not vanish from memory, and fewer active FLOPS do not guarantee proportionally lower token latency.","check":{"question":"Why can an MoE model require large memory despite low active parameters per token?","options":["All experts must remain stored or be fetched when selected","The router deletes inactive weights permanently","Active FLOPS and memory bytes are identical quantities"],"answer":0,"explanation":"Conditional execution changes which expert weights participate in a token’s work; it does not eliminate the total model state."},"science":[{"title":"Simple dispatch volume","equation":"Qdispatch = T k d s fremote","explanation":"T is tokens, k selected experts/token, d activation elements, s bytes/element and fremote the remote fraction. Q is bytes.","assumptions":"Counts one-way activation payload only; return data, metadata, duplicates and protocol overhead are additional."}],"prerequisites":["moe","parallelism"],"learningObjective":"Separate MoE’s active computation savings from total state and routing costs.","workstream":"W11","visualFamily":"model-workload","engineeringExample":"Synthetic dispatch: 1,000 tokens, two experts per token, 4,096-element activations, two bytes/element and half remote gives 8,192,000 remote payload bytes one way, before return outputs or metadata.","evidenceNotes":"Routing policy and storage placement are model-specific. The toy traffic estimate does not describe any named production model or guarantee expert balance.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}