Mixture of Experts (MoE) Explained: How AI Models Save Compute

Mixture of Experts, or MoE, is a neural network design in which a model is split into many smaller "expert" sub-networks, and a lightweight router picks only a few of them to handle each piece of input instead of running the entire model every time. The payoff is that an MoE model can hold far more total parameters than a conventional "dense" model of the same compute budget, because most of those parameters sit idle for any given token. That tradeoff — more capacity, less work per step — is why MoE has become the default way several of today's largest language models are built.

What is Mixture of Experts (MoE)?

Mixture of Experts is an architecture, not a single model — a way of structuring a neural network so that different parts of it specialize and only the relevant parts get used at any one time. The idea is decades old: the Wikipedia entry on mixture of experts traces it back to early-1990s "committee machine" research, where multiple small expert networks divided up a problem and a gating network decided how much weight to give each one's answer. What changed in the last decade is scale: once researchers combined this idea with the transformer architecture that underpins modern language models, MoE became a practical way to build enormous models that remain affordable to run.

In a transformer-based MoE model, the part that gets replaced is usually the feed-forward network (FFN) inside each transformer block — the sub-layer that processes each token's representation after attention has mixed information across the sequence. Instead of one FFN, the layer now contains several FFNs, called experts, each with its own set of weights. A small routing network, trained alongside everything else, looks at each token and decides which expert or experts should process it. The attention mechanism itself is left alone; only the feed-forward step becomes "sparse."

Dense models vs. sparse MoE models

A conventional large language model is "dense": every parameter in the network does some work on every single token that passes through it. If a dense model has 70 billion parameters, processing one token touches all 70 billion of them. That's simple and predictable, but it means compute cost scales directly with parameter count — a bigger (and presumably smarter) dense model is also a proportionally slower and more expensive one to run.

A sparse MoE model breaks that link. The model's total parameter count can be large, because it includes many experts, but the number of active parameters used for any one token is much smaller, because only a handful of experts fire. Mistral AI's Mixtral 8x7B is the most commonly cited public example: it has roughly 47 billion total parameters spread across eight experts per layer, but because its router sends each token to only two of those eight experts, it does roughly as much computation per token as a 12-13 billion parameter dense model, according to Hugging Face's technical explainer on mixture of experts and the original Mixtral of Experts paper. The attention and normalization layers are shared and used by every token regardless, which is why the "active" figure is higher than a simple 47B ÷ 4 calculation would suggest.

How MoE routing works, at a high level

The routing network is usually a very small piece of the model: a learned projection followed by a softmax, producing a score for each expert given the current token's representation. The model then keeps only the top-scoring experts — this is often called "top-k routing." Google's Switch Transformer paper simplified this further by routing each token to just a single expert (k=1), arguing that this reduces communication overhead between devices without hurting quality much, while the earlier GShard paper used top-2 routing, sending each token to two experts and blending their outputs.

Because the router is trained end-to-end with the rest of the network, it learns its own notion of what makes tokens similar — it is not told in advance that one expert should handle grammar and another should handle code. In practice, research cited by NVIDIA's technical blog on MoE has found that routing patterns are often inconsistent or idiosyncratic rather than cleanly divided by topic: the same expert pair might get picked for a recurring token like punctuation across very different subject areas, while load across experts varies by domain in ways that aren't always intuitive.

A key engineering wrinkle is load balancing. If left unchecked, a router can learn to favor a small number of "popular" experts, leaving others undertrained and effectively wasted. To prevent this, MoE training typically adds an auxiliary loss term that penalizes uneven expert usage, and systems impose a "capacity factor" — a cap on how many tokens any one expert can process per batch. Tokens that would overflow a full expert's capacity are either dropped (passed through unchanged via a residual connection) or rerouted, which is part of why getting MoE training stable is harder than training an equivalent dense model.

Why AI labs use MoE: the efficiency case

The core argument for MoE is a compute-efficiency one. Under a fixed training or inference budget, a lab can choose between (a) a dense model sized to fit that budget, or (b) a sparse MoE model with many more total parameters that still fits the same per-token compute budget because most of those parameters are inactive at any moment. Google reported that its Switch Transformer reached the same quality as a dense T5-XXL model roughly four times faster in pretraining, while scaling up to trillions of total parameters, per the Switch Transformer paper linked above.

The same logic carries into inference. Running Mixtral 8x7B costs close to what running a 12-13B dense model costs, even though Mixtral has access to the combined "knowledge" spread across all eight of its experts. That makes MoE attractive wherever serving cost and latency at scale matter — which is most commercial AI deployment. It is also one reason labs can keep growing total model capacity without every upgrade requiring a proportional jump in the GPU fleet needed to serve it.

This efficiency angle matters more as AI systems take on multi-step, tool-using work rather than single replies — the kind of agentic workflows described in Pandromeda's explainer on what MCP is and how it connects AI models to tools. Lower per-token cost means a model can afford to take more reasoning steps, call more tools, or serve more users, for the same underlying hardware.

Real-world MoE models worth knowing

MoE's path from academic curiosity to production architecture runs through a short list of well-documented systems:

• GShard (2020): Google's early demonstration that sparsely-gated MoE layers could scale a multilingual translation transformer past 600 billion parameters using automatic model sharding across TPU pods, detailed in the GShard paper above.
• Switch Transformer (2021): Simplified routing to a single expert per token and pushed total parameter counts toward the trillion-parameter range while keeping per-token compute roughly constant.
• Mixtral 8x7B (2023): Mistral AI's release put sparse MoE into a widely used, openly licensed model, pairing eight experts per layer with top-2 routing. Mistral has continued building on sparse architectures in its newer flagship releases, which Pandromeda compared directly in Mistral Large 4 vs. Claude Opus 5.5.
• DeepSeek's MoE variants: Added "shared experts" that are always active alongside routed experts, an approach Wikipedia's mixture-of-experts entry describes as a way to guarantee a baseline of general capability even when routing sends a token to narrower specialists.

MoE is not limited to text models, either. The same sparse-routing principle — specializing sub-networks and activating only what's needed — shows up in other large-scale systems being built today, including some of the real-time world-modeling approaches covered in Pandromeda's look at Genie 4, DeepMind's real-time AI world model.

The tradeoffs and open challenges

MoE is not a free lunch, and the technical sources are fairly consistent about where the costs land:

• Memory footprint: Every expert has to be loaded into memory even though only a few run per token, so a sparse MoE model needs VRAM closer to its total parameter count than its active parameter count — Mixtral needs memory comparable to a dense ~47B model despite running like a ~13B one.
• Load-balancing complexity: Keeping token traffic even across experts requires auxiliary losses and capacity limits, and real-world routing still ends up uneven — NVIDIA's analysis found some experts receiving 40-60% more tokens than others even with balancing in place, with larger imbalances on topic-concentrated inputs.
• Training and fine-tuning difficulty: Hugging Face's explainer notes that sparse models have historically been more prone to overfitting on small fine-tuning datasets than dense models of comparable active size, and that instruction-tuning tends to help MoE models by a larger margin — implying they start from a less forgiving baseline.
• Communication overhead: When experts are spread across multiple GPUs or machines (expert parallelism), tokens have to be physically routed to wherever their chosen expert lives, which adds a networking cost that a single dense model running on one device doesn't have.
• Not a guaranteed quality win: Some research cited by NVIDIA suggests that, token-for-token, a fully dense model can outperform an MoE model of similar size trained on the same amount of data — the MoE's advantage is specifically a compute-efficiency one, not an automatic quality one.

So when does MoE actually make sense?

As a rule of thumb drawn from the sources above: MoE tends to pay off in high-throughput, multi-GPU serving environments where the goal is maximizing model capacity per dollar of compute, and where the engineering team can absorb the added complexity of routing and load balancing. Dense models remain the simpler choice for smaller deployments, memory-constrained hardware, or teams that need straightforward fine-tuning behavior without the quirks sparse routing can introduce. That's also why most labs treat MoE as one architectural tool among several rather than a universal replacement for dense transformers — it solves a specific problem (more capacity without proportionally more compute) and brings a specific set of new problems with it (memory, balancing, and engineering overhead).

What's next: the practical takeaway

For anyone reading model cards or technical reports, MoE explains a pattern that otherwise looks confusing: why a model can be advertised with a huge total parameter count while behaving, in cost and speed, much closer to a far smaller model. Look for two numbers whenever a vendor describes an MoE system — total parameters and active parameters per token — because the gap between them is the entire point of the architecture. As more frontier labs publish technical reports, expect routing strategies (how many experts, how many chosen per token, whether any experts are always-on "shared" experts) to keep evolving, since that routing design is where most of MoE's remaining efficiency and stability gains are likely to come from.

Written byTheo Park

Theo Park runs the AI desk at Pandromeda. He follows model launches from the frontier labs and the open-weight community, tracks the assistants and developer tools built on them, and explains what each release changes on pricing, capability and safety. His reporting leans on primary sources: model cards, technical reports, API documentation and the companies' own announcements.

More from Theo ParkAll AI coverage