MoE experts: resident capacity, routing and misses

The mechanism

A mixture-of-experts model selects only some experts for each token, but the full parameter set still needs a valid storage and delivery strategy. A larger batch can touch many distinct experts even when each token chooses only a few. Frequently used experts may remain in fast memory; inactive expert weights can occupy another tier if the runtime can bring them in before use. Router outcomes depend on the current hidden state, so advance prediction can be imperfect. Expert parallelism also moves token activations across devices, which is a different traffic stream from fetching missing expert weights.

1. Separate shared parameters, the full expert inventory and per-token selected experts.
2. Measure how many distinct expert weights a batch touches and which remain cached.
3. Account for weight misses, staging capacity and latency; track prefetch precision and wasted bytes.
4. Route token activations to resident experts using the actual expert-parallel topology.
5. Evaluate load imbalance, missed deadlines and quality-preserving routing behavior alongside throughput.

Science and assumptions

Engineering example

Synthetic example: four distinct missed experts of 2 GiB each require 8 GiB of weight transfer at the stated precision. Ten token assignments to the same cached expert do not require ten new copies of its weights, although token dispatch still moves data.

Distinctions and tradeoffs

The union of selected experts across a batch can be much larger than the per-token selection count.

Caching popular experts saves movement but occupies scarce fast-tier space; inaccurate prefetch can consume bandwidth needed for actual misses.

MoE capacity claims become meaningful only when routing, memory placement and interconnect scheduling can deliver selected work on time.

Specifications and interfaces

Two traffic streams

Expert weights and token dispatch

They may use different paths and cannot be conflated.

Sparse activity

Few experts per token

A batch may still touch much of the expert inventory.

Interfaces

Router output Input

Selected experts and token assignments

Expert execution Output

Delivered weights, routed activations and combined output

Check your understanding

Evidence and scope

DeepSpeed supports the distinction between expert distribution and communication scheduling. HBF-based expert caching here is an explicit architectural scenario, not a feature claimed for that runtime.

Place read-only weights and mutable KV across specified tiers and inspect transfer paths; no HBF shipping-system benchmark or universal memory semantics are implied.

Sources

• DeepSpeed — MoE inference
• Sandisk — release of first OCP HBF specification

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• MoE routing, active parameters and resident state
• Place the working set, not just the model
• HBF latency, transfer granularity and prefetch

Related lessons

• Collective traffic and congestion
• From memory placement to network, power and cooling
• Communication is the realization medium