RAMP

A new Tencent paper finds that domain skills such as math and coding in Mixture-of-Experts models are not concentrated in individual experts but scattered across sparse neurons spanning multiple experts. Neuron-level pruning scored by routed contribution (RAMP) retains 91%-98% of performance with 75% of neurons kept; at 50% retention, GPU memory drops 36%-43% and batched prefill throughput rises 17%-39%, offering an alternative to whole-expert pruning for MoE compression.