Mixture-of-Experts Routing Adds a Second Layer of Inference Variation

Mixture-of-Experts models such as Mixtral 8x7B¹ replace the dense feed-forward block with parallel “expert” subnetworks. A small router picks which experts process each token. In Mixtral’s case, each token goes to 2 of 8 experts per layer.

Capacity is large; active parameters per token are smaller (~47B total, ~13B active for Mixtral¹). That is the efficiency argument. It is also a reconstruction problem.

Routing variation

The router is itself a learned network: a distribution over experts per token.

Context-sensitive routing. A small change in the prompt can shift expert choices across layers. The map from input perturbation to routing change is nonlinear and hard to predict from the outside.

Approximate load balancing. Training uses auxiliary losses to encourage balanced expert use, but published work still reports large distributional imbalances². Some experts overload; others idle.

Combine that with floating-point nondeterminism on GPUs and two “identical” runs can activate different experts and then compute slightly different results inside those experts.

Which parameters ran

In a dense model, every parameter participates in every forward pass. The mapping is complex, but the parameter set is fixed. In MoE, the parameter set that produced an output is itself variable.

That raises three review questions:

1. Which parameters contributed? Dense: all of them. MoE: a subset chosen by routing. Explaining an output requires expert assignments, not only the input.
2. Can the run be reconstructed? Without logged routing, input and weights are not enough. Reviewers need per-token, per-layer expert IDs — data standard inference pipelines usually discard.
3. How sensitive is the output to routing? Work on routing robustness³ finds standard routers can pick suboptimal experts on out-of-distribution inputs; when inputs leave the training distribution, the router extrapolates from fixed priors.

MoE under review

If MoE is on the table for a governed deployment:

Capture routing with the run. Replay and audit need expert assignments alongside input and output. That is logging overhead dense models do not require.

Treat architecture choice as a governance decision. MoE vs dense changes what can be reconstructed, what can be explained, and what variation you accept. Make that tradeoff before production — not after an auditor asks.

Dense is simpler to govern when reconstruction is the priority. Hardware-level nondeterminism still applies, but “which parameters contributed?” has a fixed answer.

MoE architectures are effective and increasingly common. Routing variability is not a reason to ban them. It is a reason to instrument them before the first production inference you may need to defend.

References