The mechanism
A mixture-of-experts layer sends each token to a selected subset of expert networks. This can increase total model parameters without activating all of them for every token. The inactive experts still need storage somewhere, and tokens must reach the devices holding their selected experts. Routing imbalance creates overloaded experts, padding or dropped work depending on the design. Small per-expert batches may also use matrix units inefficiently. The sparsely gated MoE research established conditional computation and load-balancing concerns. A useful resource card therefore records total parameters, active parameters, routing policy and communication rather than replacing model size with one active-parameter number.
- Calculate router scores and select the permitted experts for each token under the model’s policy.
- Group tokens by destination expert and check expert capacity or load-balancing constraints.
- Dispatch activations to the devices holding those experts, including remote transfers where necessary.
- Execute the resulting expert batches and account for padding, stragglers or sparse utilization.
- Return and combine outputs, then measure total state, active arithmetic and communication as separate budgets.
Science and assumptions
Simple dispatch volume
Remote dispatch bytes equal tokens times experts per token times activation width times bytes per element times the remote-dispatch fraction.
Symbols and units
- Q_dispatch
- One-way remote activation payload
bytes
- T
- Tokens dispatched
tokens
- k
- Selected experts per token
experts/token
- d
- Activation width copied per dispatch
elements/dispatch
- s
- Stored activation element size
bytes/element
- f_remote
- Fraction of expert dispatches crossing the selected boundary
fraction
T is tokens, k selected experts/token, d activation elements, s bytes/element and fremote the remote fraction. Q is bytes.
Assumptions Counts one-way activation payload only; return data, metadata, duplicates and protocol overhead are additional.
Engineering example
Synthetic dispatch: 1,000 tokens, two experts per token, 4,096-element activations, two bytes/element and half remote gives 8,192,000 remote payload bytes one way, before return outputs or metadata.
Distinctions and tradeoffs
Inactive parameters do not vanish from memory, and fewer active FLOPS do not guarantee proportionally lower token latency.
Sparse activation saves arithmetic relative to executing every expert, but adds routing, load imbalance, fragmented batches and potentially large remote memory traffic.
MoE can shift a bottleneck from dense arithmetic toward expert storage and all-to-all communication without reducing all system costs in proportion to active parameters.
Specifications and interfaces
- Two parameter counts
- Resident total and active per token
Shared layers and routing overhead must also be included.
- Routing traffic
- Tokens × selected experts × activation bytes
Remote fraction and return traffic depend on placement.
Interfaces
- Token activations and router Input
- Expert choices, weights and capacity decisions
- Expert outputs Output
- Dispatched, computed and combined token results
Check your understanding
Why can an MoE model require large memory despite low active parameters per token?
- All experts must remain stored or be fetched when selected
- The router deletes inactive weights permanently
- Active FLOPS and memory bytes are identical quantities
Answer and reasoning
All experts must remain stored or be fetched when selected
Conditional execution changes which expert weights participate in a token’s work; it does not eliminate the total model state.
Evidence and scope
Routing policy and storage placement are model-specific. The toy traffic estimate does not describe any named production model or guarantee expert balance.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.