MarkTechPost compared how open Mixture-of-Experts models route tokens. They all do token-choice routing, but almost every knob is different.
- Expert counts from 8 to 384; top-k from 1 to 8; some keep a shared expert, some don’t.
- Routed sparsity spans about 4x (Mixtral) to 128x (Llama 4 Maverick).
- Kimi K2 found sparsity 48 needed 1.69x fewer FLOPs than sparsity 8 at equal validation loss.
- Llama 4 Maverick and gpt-oss don’t publish their gating method in config.json.
My take
“17B active” hides a lot. Read the router config before you pick a model for a workload.
God Bless, and May the Source Be with You!
