Expert Pruning

An Amazon research team introduces HOPE, a higher-order expert pruning method for Mixture-of-Experts LLMs that removes redundant experts by accounting for inter-expert coupling rather than scoring them independently. The work targets the memory bottleneck where expert MLPs dominate inference footprint: Qwen3.5-122B has 256 experts per layer across 48 MoE layers, with 95% of its ~250 GB parameters in expert MLPs while only 8 experts per layer activate per token.