# MoE experts: resident capacity, routing and misses

Canonical: https://semiaifoundry.com/learn/atlas/moe-expert-tier-placement/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Software · Atlas v1.3.0

# MoE experts: resident capacity, routing and misses

Explain why sparse activation reduces arithmetic without eliminating expert storage or transfer constraints.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#moe-expert-tier-placement)Router decisions, expert residency and physical link traffic; abstraction

Learning objectiveExplain why sparse activation reduces arithmetic without eliminating expert storage or transfer constraints.

## The mechanism

A mixture-of-experts model selects only some experts for each token, but the full parameter set still needs a valid storage and delivery strategy. A larger batch can touch many distinct experts even when each token chooses only a few. Frequently used experts may remain in fast memory; inactive expert weights can occupy another tier if the runtime can bring them in before use. Router outcomes depend on the current hidden state, so advance prediction can be imperfect. Expert parallelism also moves token activations across devices, which is a different traffic stream from fetching missing expert weights.

1.  Separate shared parameters, the full expert inventory and per-token selected experts.
2.  Measure how many distinct expert weights a batch touches and which remain cached.
3.  Account for weight misses, staging capacity and latency; track prefetch precision and wasted bytes.
4.  Route token activations to resident experts using the actual expert-parallel topology.
5.  Evaluate load imbalance, missed deadlines and quality-preserving routing behavior alongside throughput.

## Science and assumptions

### Expert-weight miss traffic

$$
V_{\mathrm{miss}}=\sum_{e\in\mathcal{M}}P_e b_e
$$

Vmiss = Σe∈M Pe be

Symbols and units

Vmiss

Transferred expert-weight bytes

bytes

M

Distinct expert cache misses in the window

set

Pe

Parameters in missing expert e

parameters

be

Stored bytes per parameter

bytes/parameter

The set M contains distinct missing experts required during the measured reuse window; count each transferred expert once in that window.

Assumptions Whole expert transfer at specified stored precision, no shared/sub-tensor fetch or compression metadata; streaming changes the model.

## Engineering example

Synthetic example: four distinct missed experts of 2 GiB each require 8 GiB of weight transfer at the stated precision. Ten token assignments to the same cached expert do not require ten new copies of its weights, although token dispatch still moves data.

## Distinctions and tradeoffs

The union of selected experts across a batch can be much larger than the per-token selection count.

Caching popular experts saves movement but occupies scarce fast-tier space; inaccurate prefetch can consume bandwidth needed for actual misses.

MoE capacity claims become meaningful only when routing, memory placement and interconnect scheduling can deliver selected work on time.

## Specifications and interfaces

Two traffic streams

Expert weights and token dispatch

They may use different paths and cannot be conflated.

Sparse activity

Few experts per token

A batch may still touch much of the expert inventory.

### Interfaces

Router output Input

Selected experts and token assignments

Expert execution Output

Delivered weights, routed activations and combined output

## Check your understanding

### Each token selects two experts. Must a batch need only two experts resident?

1.  No: tokens can choose different experts
2.  Yes: top-2 means only two experts exist
3.  Yes: unused experts have no stored weights

Answer and reasoning

No: tokens can choose different experts

The union of selected experts across a batch can be much larger than the per-token selection count.

## Evidence and scope

DeepSpeed supports the distinction between expert distribution and communication scheduling. HBF-based expert caching here is an explicit architectural scenario, not a feature claimed for that runtime.

Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.

## Sources

-   [DeepSpeed — MoE inference](https://www.deepspeed.ai/tutorials/mixture-of-experts-inference/)
-   [Sandisk — release of first OCP HBF specification](https://investor.sandisk.com/node/8126/pdf)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/moe-expert-tier-placement-d2d76a649e40c474.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [MoE routing, active parameters and resident state](https://semiaifoundry.com/learn/atlas/expert-routing-budget/)
-   [Place the working set, not just the model](https://semiaifoundry.com/learn/atlas/memory-working-set-placement/)
-   [HBF latency, transfer granularity and prefetch](https://semiaifoundry.com/learn/atlas/hbf-latency-prefetch/)

### Related lessons

-   [Collective traffic and congestion](https://semiaifoundry.com/learn/atlas/fabric-collective-traffic/)
-   [From memory placement to network, power and cooling](https://semiaifoundry.com/learn/atlas/memory-to-hall-propagation/)
-   [Communication is the realization medium](https://semiaifoundry.com/learn/atlas/crg-communication-graph/)