Interactive Lab · Mixture-of-Experts

Don't run the whole model, route each token to a few experts

A giant model is expensive because every token flows through every parameter. Mixture-of-Experts makes it cheap: split the layer into many small experts, and let a tiny gating network send each token to just its best top-k. You get the capacity of a huge model at the cost of a small one, if the router behaves. Left alone it plays favourites, cramming everything into a few experts while the rest sit idle. Watch it happen, then slide load-balancing on.

▶ The playground, route the tokens

experts 6 top-k 1 load min/max experts used status imbalanced

routed (top-k) · brighter = higher gate weight overloaded / dead expert

Experts (N) 6

Top-k (experts per token) 1

1 · switch3 · dense-ish

Load balancing 0.0

0 · raw gate (plays favourites)balanced · 1

🧭 Where this fits

Mixture-of-Experts is how frontier models (Mixtral, DeepSeek, GPT-class) get huge parameter counts without huge cost: route each token to a few expert sub-networks. It's why a “trillion-parameter” model can run affordably. Reach for it to scale capacity while keeping compute-per-token flat.

The code that's actually running

🔮 Predict, then test

Set load balancing to 0 and look at the load bars: a couple of experts are red and overloaded, and at least one is dead, it never gets a token, so it never learns, so it stays useless (a vicious cycle called expert collapse). Now slide balancing toward 1 and watch the tokens spread until every expert is carrying its share. Real MoE models add exactly this, an auxiliary load-balancing loss, because without it, a trillion-parameter model quietly behaves like a much smaller one. Also try top-k = 1 vs 2: more experts per token = more compute but smoother routing.

The takeaway

Your new mental model of Mixture-of-Experts

Sparse activation. Each token uses only k of N experts, so you can scale total parameters way up while keeping the compute-per-token flat. That's the whole appeal.
A gate decides. A tiny network scores the experts for each token and routes to the top-k, a learned, per-token dispatch.
Balance is the hard part. A raw gate collapses onto a few experts (rich get richer, others die). An auxiliary load-balancing loss spreads the tokens so all the capacity gets used.
It's how frontier models scale. Mixtral, DeepSeek, GPT-class MoEs, huge parameter counts, but each token only pays for a handful of experts.

The gate's softmax →Attention 📖 Lecture notes