Mixture of experts (MoE)

A neural network design in which a router sends each token to a few specialist sub-networks, so only part of the model runs for each step.

MoE models can hold hundreds of billions of parameters while computing each token with only a fraction of them. Memory must still fit all experts.

#Glossary #Architecture

Stories about Mixture of experts (MoE)

Paper of the DayResearch

Paper of the day: Switch Transformers and the rise of mixture-of-experts

Switch Transformers showed that routing each token to one expert lets models grow to a trillion parameters without proportional compute. How MoE works and why it matters.

AI DeskOct 184 min read