Mixture of Experts
A model architecture that routes each input through selected expert subnetworks instead of activating the whole model.
Plain English
Only the most relevant parts of the model wake up for a given token.
Example
A sparse model can have many total parameters while using only a subset during inference.
Why it matters
MoE is one way frontier labs improve capability while managing inference cost.