Mixture of Experts

Route each token to a few specialists

A mixture-of-experts model holds many internal sub-networks. For each token a small gating router picks a handful, their outputs are combined, and everything else stays switched off.

The point is the ratio between two numbers. Total capacity can be enormous while the compute spent on any single token stays modest, which is how a model can be very large and still affordable to run.

Two things are commonly misread. The experts are not tidy human categories — nobody assigned one to law and another to chemistry, and what each has specialized in is usually not describable. And routing happens per token, not per request, so one sentence can touch many different experts on its way through.