A mixture of experts (MoE) model is a sparse network whose feedforward block picks from distinct groups of parameters, the experts. At every layer, for every token, a router network chooses a small number of those groups to process the token, according to Mistral AI.
The name misleads. An expert here is a block of weights inside a neural network, separate from any human specialist and from the ensemble methods that older statistical literature labels the same way. The design goes back to a Sparsely-Gated Mixture-of-Experts layer consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of those experts to use for each example.
Key Takeaways
- In Mixtral, the feedforward block picks from a set of 8 distinct groups of parameters, and a router network chooses two of those groups to process each token at every layer.
- Mixtral has 46.7 billion total parameters but only uses 12.9 billion parameters per token. It processes input and generates output at the same speed and for the same cost as a 12.9 billion parameter model.
- Llama 4 Maverick contains 17 billion active parameters, 128 experts, and 400 billion total parameters, according to Meta AI.
- DeepSeek-V3 carries 671 billion total parameters with 37 billion activated for each token, per DeepSeek-AI.
- MoE layers in Meta’s Llama 4 design use 128 routed experts and a shared expert. Each token is sent to the shared expert and also to one of the 128 routed experts.
- All parameters are stored in memory, while only a subset of the total parameters are activated while serving these models. That improves inference efficiency by lowering model serving costs and latency.
How Does a Mixture of Experts Model Work?
Two moving parts carry the design: a set of expert sub-networks, and a router that picks between them. The technique increases the number of parameters of a model while controlling cost and latency, per Mistral AI. The model only uses a fraction of the total set of parameters per token. Everything else in the stack stays as it is across large language model architectures.
1. The feedforward block splits into experts
Each layer is composed of 8 feedforward blocks, the experts, according to Mixtral’s technical report. The attention machinery is untouched. Only the dense feedforward slab is cut into parallel copies, each holding its own weights.
A warehouse makes the shape concrete. The full catalog stays on the shelves, and a picker walks to only a few bins to fill any single order. Shelf space is the memory bill; walking distance is the compute bill.
2. The router scores every token
A trainable gating network determines a sparse combination of experts to use for each example, which is the mechanism the sparsely-gated layer introduced. Routing is learned during training rather than configured by hand. Each token sees two experts, and the selected experts can be different at each timestep.
3. Only the selected experts run
Llama 4 alternates dense and mixture-of-experts layers for inference efficiency, per Meta AI. Its MoE layers use 128 routed experts plus a shared expert. The shared expert runs on everything. The routed pool is where the sparsity lives, since a token touches one member of it and skips the rest.
4. The outputs combine and pass on
A router network chooses two of the expert groups to process the token and combines their output additively. The merged vector then moves to the next layer, where the routing decision is made again from scratch.
A triage desk is the closer analogy. The desk reads each case and forwards it to two specialists rather than to every clinician in the building. The building still has to be staffed and lit whether or not a given specialist sees anyone that day.
| Component | What it does | Example from a shipped model |
|---|---|---|
| Expert | A feedforward sub-network holding its own block of parameters | Mixtral splits each feedforward block into 8 experts |
| Router (gating network) | Scores every token and selects which experts run | Mixtral’s router picks 2 experts per token per layer |
| Shared expert | Runs on every token regardless of the routing decision | Llama 4 sends each token to a shared expert |
| Routed experts | The pool the router selects from | Llama 4 MoE layers hold 128 routed experts |
| Combination step | Merges the selected experts’ outputs | Mixtral combines the two outputs additively |
Sources: Mistral AI, Meta AI
Those two moving parts produce a number that model tables almost always report wrong.
Active Parameters vs Total Parameters in MoE Models
Mixtral has 46.7 billion total parameters but only uses 12.9 billion parameters per token, according to Mistral AI. The technical report states the same split with rounded figures. Each token has access to 47 billion parameters but only uses 13 billion active parameters during inference.
Llama 4 Maverick contains 17 billion active parameters, 128 experts, and 400 billion total parameters, per Meta AI. Llama 4 Scout has 17 billion active parameters, 16 experts, and 109 billion total parameters. DeepSeek-V3 carries 671 billion total parameters with 37 billion activated for each token.
Every one of those announcements publishes an active figure and a total figure, and none of them divides one by the other. The quotient is what tells a reader how much of the model actually runs.
Mixtral activates 27.6% of its parameter count on any given token. Llama 4 Scout drops to 15.6%. DeepSeek-V3 runs at 5.5%. Llama 4 Maverick sits at 4.3%.
The ordering runs opposite to intuition. The bigger the headline parameter count, the smaller the share doing work on any single token. A specification cell reading 400 billion parameters describes storage rather than per-token work. A side-by-side model specification tracker needs both figures in the same row.
| Model | Active parameters per token (billions) | Total parameters (billions) | Active share |
|---|---|---|---|
| Mixtral | 12.9 | 46.7 | 27.6% |
| Llama 4 Scout | 17 | 109 | 15.6% |
| DeepSeek-V3 | 37 | 671 | 5.5% |
| Llama 4 Maverick | 17 | 400 | 4.3% |
Sources: Mistral AI, Meta AI, DeepSeek-AI
All parameters are stored in memory, while only a subset of the total parameters are activated while serving these models. So the total count still sets the memory requirement even when it stops setting the compute requirement. The comparison only pays off once it is set against the architecture that MoE replaced.
Mixture of Experts vs Dense Models
In deep learning, models reuse the same parameters for all inputs. Mixture of experts instead selects different parameters for each incoming example. The result is a sparsely-activated model with outrageous numbers of parameters but a constant computational cost. That single difference reshapes the cost curve.
MoE architectures are more compute efficient for training and inference, according to Meta AI. Given a fixed training FLOPs budget, they deliver higher quality compared to a dense model. Quality at a fixed budget is a different axis from factual reliability, which LLM hallucination rate data measures on its own terms.
The cost lands elsewhere, since widespread adoption of MoE has been hindered by complexity, communication costs, and training instability.
| Dimension | Dense model | MoE model |
|---|---|---|
| Parameters used per input | Every parameter, on every input | Only the experts the router selects |
| Parameter count reachable | Bounded by the compute budget | Outrageous numbers at constant computational cost |
| Memory while serving | Matches the parameter count | Matches the total count, not the active count |
| Training profile | Established and stable | Complexity, communication costs and training instability |
| Quality per fixed training FLOPs budget | Lower | Higher, per Meta AI |
Sources: Meta AI, Switch Transformers research
Are MoE models cheaper to run?
Sparse activation improves inference efficiency by lowering model serving costs and latency, while all parameters remain stored in memory, per Meta AI. The memory bill and the compute bill move separately here. Cheaper per token does not translate into cheaper to host, because the weights still have to sit somewhere.
Why Does Mixture of Experts Matter?
Conditional computation means parts of the network are active on a per-example basis. It has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. The sparsely-gated layer achieved greater than 1000x improvements in model capacity, with only minor losses in computational efficiency on modern GPU clusters. It used an MoE of up to 137 billion parameters applied convolutionally between stacked LSTM layers.
The Switch Transformer simplified the MoE routing algorithm and obtained up to 7x increases in pre-training speed with the same computational resources. It then pre-trained up to trillion-parameter models and achieved a 4x speedup over the T5-XXL model. Speed at a fixed budget is what moved the idea out of the research literature.
The largest GLaM model has 1.2 trillion parameters, approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference. Those savings land on hardware budgets, which track AI chip shipment data more closely than they track any single architecture choice.
Our AI model coverage keeps surfacing the same lag. Capability rankings move every few release cycles, while the public shorthand for model size stays frozen at whichever figure an announcement led with. Model size stopped being a single number, so every capability comparison now needs two.
Pros, Cons, and Risks of Mixture of Experts
A design that changes what a parameter count means also changes what can go wrong.
Advantages
- Model capacity increases without a proportional increase in computation.
- Parameter counts rise to outrageous numbers while the computational cost stays constant.
- Compute efficiency improves for training and inference. A fixed training FLOPs budget delivers higher quality than a dense model.
- Activating only a subset of the total parameters lowers model serving costs and latency.
Trade-offs and Risks
- Complexity, communication costs, and training instability have hindered widespread adoption.
- All parameters are stored in memory while serving, even though only a subset is activated.
- DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing.
- Expert routing strategies that have cross-batch dependencies are vulnerable to attacks.
Cross-batch routing risk: Malicious queries can be sent to a model and can affect a model’s output on other benign queries if they are grouped in the same batch. Routing designs without cross-batch dependencies help reduce that exposure, and no routing choice removes it.
| Risk | What it looks like | Mitigation direction |
|---|---|---|
| Training instability | Complexity and communication costs during training | Simplified routing algorithms |
| Memory footprint | Every parameter resident while serving | Capacity planning against the total count |
| Expert load imbalance | Traffic concentrates on part of the expert pool | Explicit load-balancing strategies |
| Cross-batch routing | One query influences another query’s output in the same batch | Routing without cross-batch dependencies |
Sources: Meta AI, DeepSeek-AI, Google DeepMind research
Routing is one of several ways an input can shift a model’s behavior. Training-time manipulation is a separate mechanism, tracked in training data poisoning research.
Real-World Applications of Mixture of Experts
Three shipped model families show the design at different scales, and each publishes its own figures.
Open-weight frontier models
Mixtral is a decoder-only sparse mixture-of-experts network whose feedforward block picks from a set of 8 distinct groups of parameters, per Mistral AI. It has 46.7 billion total parameters and only uses 12.9 billion per token. Open weights at that ratio put a frontier-scale model on hardware sized for a much smaller one.
Multimodal model families
Llama 4 models are the first Meta models that use a mixture-of-experts architecture, and they alternate dense and mixture-of-experts layers for inference efficiency. Llama 4 Scout supports a context length of 10 million tokens, according to Meta AI.
Very large open reasoning models
DeepSeek-V3 is a Mixture-of-Experts language model with 671 billion total parameters and 37 billion activated for each token. It was pre-trained on 14.8 trillion diverse and high-quality tokens and required only 2.788 million H800 GPU hours for its full training. The training-cost figure is the part that made the release notable.
Scenario: One token moves through an MoE layer
- The token arrives at the layer as a vector carried forward from the attention block.
- The router network chooses two of the expert groups.
- Only those two experts run their feedforward computation; the remaining groups stay idle in memory.
- Their outputs combine additively. The combined vector then passes to the next layer.
- At the next timestep, the router can select different experts for the same sequence.
Every one of those deployments inherits the same open question about the router.
Is Mixture of Experts More Secure Than a Dense Model?
No, and the architecture adds a surface that a dense model does not have. Expert routing strategies that have cross-batch dependencies are vulnerable to attacks. Malicious queries can affect a model’s output on other benign queries if they are grouped in the same batch. That result comes from a proof-of-concept attack in a toy experimental setting.
The scoping matters. The work establishes a mechanism under laboratory conditions, and prevalence in deployed systems remains unmeasured.
All parameters are stored in memory while only a subset activates during serving. Sparse activation shrinks the compute per token and leaves the resident weight set where it was. Guardrail bypass is measured on a separate axis, in LLM jailbreak success rate data.
Conclusion
Mixtral holds 46.7 billion total parameters and only uses 12.9 billion per token, one instance of the total-versus-active split these architectures turn on. That share, 27.6%, is what a single parameter count hides. Llama 4 Maverick widens the same gap to 17 billion active against 400 billion total, according to Meta AI.
The practical reading for anyone comparing models is that one figure no longer describes a system. One number sets the memory bill, the other sets the compute bill, and on current releases the two differ by more than an order of magnitude. Any comparison that quotes only the larger of the two is describing storage.
