# MoE routing, active parameters and resident state

Canonical: https://semiaifoundry.com/learn/atlas/expert-routing-budget/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Model–machine co-design · Atlas v1.3.0

# MoE routing, active parameters and resident state

Balances conditional expert computation against placement, communication and capacity.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#expert-routing-budget)Accelerator group to serving pool; routes schematic

Learning objectiveSeparate MoE’s active computation savings from total state and routing costs.

## The mechanism

A mixture-of-experts layer sends each token to a selected subset of expert networks. This can increase total model parameters without activating all of them for every token. The inactive experts still need storage somewhere, and tokens must reach the devices holding their selected experts. Routing imbalance creates overloaded experts, padding or dropped work depending on the design. Small per-expert batches may also use matrix units inefficiently. The sparsely gated MoE research established conditional computation and load-balancing concerns. A useful resource card therefore records total parameters, active parameters, routing policy and communication rather than replacing model size with one active-parameter number.

1.  Calculate router scores and select the permitted experts for each token under the model’s policy.
2.  Group tokens by destination expert and check expert capacity or load-balancing constraints.
3.  Dispatch activations to the devices holding those experts, including remote transfers where necessary.
4.  Execute the resulting expert batches and account for padding, stragglers or sparse utilization.
5.  Return and combine outputs, then measure total state, active arithmetic and communication as separate budgets.

## Science and assumptions

### Simple dispatch volume

$$
Q_{\mathrm{dispatch}}=T k d s f_{\mathrm{remote}}
$$

Remote dispatch bytes equal tokens times experts per token times activation width times bytes per element times the remote-dispatch fraction.

Symbols and units

Q\_dispatch

One-way remote activation payload

bytes

T

Tokens dispatched

tokens

k

Selected experts per token

experts/token

d

Activation width copied per dispatch

elements/dispatch

s

Stored activation element size

bytes/element

f\_remote

Fraction of expert dispatches crossing the selected boundary

fraction

T is tokens, k selected experts/token, d activation elements, s bytes/element and fremote the remote fraction. Q is bytes.

Assumptions Counts one-way activation payload only; return data, metadata, duplicates and protocol overhead are additional.

## Engineering example

Synthetic dispatch: 1,000 tokens, two experts per token, 4,096-element activations, two bytes/element and half remote gives 8,192,000 remote payload bytes one way, before return outputs or metadata.

## Distinctions and tradeoffs

Inactive parameters do not vanish from memory, and fewer active FLOPS do not guarantee proportionally lower token latency.

Sparse activation saves arithmetic relative to executing every expert, but adds routing, load imbalance, fragmented batches and potentially large remote memory traffic.

MoE can shift a bottleneck from dense arithmetic toward expert storage and all-to-all communication without reducing all system costs in proportion to active parameters.

## Specifications and interfaces

Two parameter counts

Resident total and active per token

Shared layers and routing overhead must also be included.

Routing traffic

Tokens × selected experts × activation bytes

Remote fraction and return traffic depend on placement.

### Interfaces

Token activations and router Input

Expert choices, weights and capacity decisions

Expert outputs Output

Dispatched, computed and combined token results

## Check your understanding

### Why can an MoE model require large memory despite low active parameters per token?

1.  All experts must remain stored or be fetched when selected
2.  The router deletes inactive weights permanently
3.  Active FLOPS and memory bytes are identical quantities

Answer and reasoning

All experts must remain stored or be fetched when selected

Conditional execution changes which expert weights participate in a token’s work; it does not eliminate the total model state.

## Evidence and scope

Routing policy and storage placement are model-specific. The toy traffic estimate does not describe any named production model or guarantee expert balance.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/abs/1701.06538)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/expert-routing-budget-274a075fef73b09f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Mixture of experts](https://semiaifoundry.com/learn/atlas/moe/)
-   [Data, tensor, pipeline, and state parallelism](https://semiaifoundry.com/learn/atlas/parallelism/)

### Related lessons

-   [Autoregressive decode and weight traffic](https://semiaifoundry.com/learn/atlas/decode-bandwidth-trace/)