Mixture of experts still needs a scheduler

Sparse arithmetic does not imply idle infrastructure

Conditional computation separates total model capacity from the computation activated for one input. That is an appealing idea, but the inactive parameters still have to exist somewhere, and the active ones must be reachable. A mixture-of-experts layer therefore creates a routing and placement problem alongside its mathematical function. I would resist describing its cost using only the number of parameters touched per token. Device memory, communication, padding, load imbalance and synchronization can determine whether the theoretical reduction in arithmetic becomes a useful reduction in request latency or a useful increase in throughput.

Fedus, Zoph and Shazeer’s Switch Transformers explores a simplified sparse routing design that selects one expert per token. Lepikhin and colleagues’ GShard studies conditional computation together with automatic sharding. Their work makes the systems connection explicit: sparse model structure and distributed execution have to be designed together. The papers’ experimental results belong to their architectures and environments. My analysis here focuses on a smaller operational consequence: even a routing decision with very little mathematical complexity can create a queue at one expert while leaving capacity unused elsewhere. A scheduler must decide what that imbalance means for the request.

References: [1] Switch Transformers — Fedus, Zoph and Shazeer[2] GShard — Lepikhin and colleagues

Work through an imbalanced batch

Imagine a top-one routing layer with one hundred and twenty-eight tokens and eight experts. With a capacity factor of 1.25, a simple capacity formula assigns each expert twenty token slots: the ceiling of 1.25 times 128 divided by eight. There are one hundred and sixty slots overall. Now suppose the router sends sixty-four tokens to the first expert, sixteen to the second and eight to each of the other six. The total demand is still one hundred and twenty-eight, but forty-four assignments exceed the first expert’s capacity while seventy-six allocated slots elsewhere remain unused.

Those figures are an illustrative accounting exercise, not a claim about the distribution of any published model. They show why total capacity is insufficient as a safety check. The system could queue excess work, choose another expert, change batch formation or use a defined overflow behavior. Each option has consequences. Queueing changes latency; rerouting changes the computation; padding spends resources on unused slots. Dropping a contribution cannot be treated as a scheduling optimization unless the model and application explicitly permit that behavior. The correct policy depends on the trained architecture and serving contract, not merely on which choice is easiest to implement.

from math import ceil
tokens, experts, factor = 128, 8, 1.25
capacity = ceil(factor * tokens / experts)
demand = [64, 16, 8, 8, 8, 8, 8, 8]
served = sum(min(n, capacity) for n in demand)
overflow = sum(max(0, n - capacity) for n in demand)
unused = experts * capacity - served
assert sum(demand) == tokens and capacity == 20
assert (served, overflow, unused) == (84, 44, 76)

Balance is a learned and an operational property

A routing objective can encourage a useful distribution of tokens during training, but that does not eliminate operational skew. The serving workload may differ from training, a burst can contain unusually similar inputs, or the available batch may be too small to average out variation. I would measure expert utilization over short intervals as well as across the whole day. A balanced daily total can coexist with repeated short periods in which one expert determines the latency of every active request. Those periods are precisely where an interactive product feels unreliable, even when overall utilization looks healthy.

The scheduler also has information that the router may not: current device load, pending requests, network contention and deadlines. Giving that information to a routing policy may seem attractive, but changing expert selection is a semantic change unless the model was designed and evaluated for it. I would first explore placement, batching and admission controls that preserve the intended computation. If load-aware rerouting is part of the design, evaluate its quality under the conditions that trigger it. Otherwise a performance emergency can silently substitute a different model behavior at the exact moment observability is already under pressure.

References: [1] Switch Transformers — Fedus, Zoph and Shazeer

Communication competes with useful computation

When experts live on different devices, token representations must move to the selected experts and their outputs must return to the appropriate sequence positions. The arithmetic saved by sparsity can be offset by transfers and synchronization, especially when individual expert batches are small. I would account for bytes moved per layer, collective communication time and the distribution of local batch sizes. A peak arithmetic throughput number does not capture these costs. The relevant path includes dispatch, execution and combination, with the slowest participating resource potentially controlling when the next dependent operation can begin.

Expert placement can improve locality but introduces another tradeoff. Replicating a popular expert reduces some transfer or contention costs while consuming memory that could hold other experts or larger caches. Moving an expert dynamically has its own transfer and warm-up cost. GShard’s combination of model structure and sharding is useful context for this problem, rather than a universal deployment recipe. My preference is to model the expected traffic matrix and then verify it with traces from the actual workload. Optimizing an imagined uniform distribution is easy; serving an uneven sequence of real requests is the test that matters.

References: [2] GShard — Lepikhin and colleagues

Interactive latency and batch throughput diverge

Larger batches can make expert execution more efficient by providing more tokens per expert, but assembling those batches may delay an individual request. That tension is familiar in dense inference and becomes more complicated when the batch fragments across experts. A token can wait for peers that will be routed elsewhere and still arrive at its expert in a small group. I would evaluate time to first useful output and the gaps between subsequent outputs, not just tokens processed per second. A configuration that wins a saturated throughput test may produce an unpleasant conversational experience at moderate or uneven load.

Deadline-aware batching should impose a maximum wait rather than accumulate work indefinitely in pursuit of efficiency. It also needs to distinguish requests that are still useful from requests whose users have cancelled. Continuing to route cancelled tokens can waste scarce expert capacity and lengthen other queues. Cancellation must respect the runtime’s synchronization requirements, so it may not interrupt an in-flight collective immediately. The important contract is that the scheduler stops admitting unnecessary future work and accounts for the work already committed. Sparse computation does not excuse the service from ordinary lifecycle and backpressure obligations.

Parameter count is an incomplete cost label

A model with a large total parameter count and relatively few active parameters can have a very different memory footprint from a dense model with the same active count. It can also have a different communication pattern from a dense model with the same total count. I would report both counts where useful, then describe the hardware arrangement and execution policy. Neither number alone establishes price, speed or quality. A reader choosing a model needs to know whether the claimed benefit depends on many devices, a particular batch size or a workload that consistently balances expert demand.

There is a reasonable counterargument that users should not need to care about these details if a provider delivers an acceptable service. At the product boundary, that is correct. The provider still needs an internal model of the constraints, and the customer still benefits from measured latency, availability and quality under their own traffic. A simple external interface can hide implementation complexity without pretending it does not exist. My objection is to using sparse activation as a guarantee of cheap or fast execution before the scheduling and placement costs have been measured in the intended environment.

Observe the queues created by the architecture

The useful dashboard is not a single average utilization chart. I want expert demand distributions, capacity overflows, dispatch and combine time, queue age, cancellation lag and quality metrics for any alternate execution path. These measurements should be correlated with request shape and workload type so that an imbalance can be explained. A spike in one expert’s demand may reveal a legitimate workload change rather than a broken router. The response should preserve the model’s behavioral contract while managing the resource problem, and the trace should make that distinction visible to the person investigating it.

The central lesson is that conditional computation relocates scheduling work; it does not remove it. Selecting fewer parameters per token is valuable when the system can exploit that structure without spending the savings on movement, waiting or semantic compromises. I would evaluate an expert model as a joint numerical and distributed system: what is selected, where it runs, how it queues and what happens when demand is uneven. That perspective makes sparse architectures more understandable and their benefits more credible, because the performance claim rests on the complete execution path rather than on a parameter-count slogan.