腾讯新研究揭示 MoE 领域技能分散于专家内部神经元
腾讯论文发现，MoE 模型中的数学、代码等领域能力并非集中在单个专家，而是分散在跨专家的稀疏神经元上。基于路由贡献度评分（RAMP）的神经元级剪枝，在保留 75% 神经元时仍维持 91%-98% 性能；保留 50% 时 GPU 显存下降 36%-43%，批量预填充吞吐提升 17%-39%。这为 MoE 压缩提供了替代整专家剪枝的新路径。
报道正文
Rohan Paul (@rohanpaul_ai) · 2026-11-13 16
New Tencent paper shows domain skills (like math or coding ) in Mixture of Experts models sit in scattered neurons across expert.
so cutting weak neurons instead of whole experts shrinks the models while keeping most skills.
In their experiment, Mixture of Experts AI models kept 91% to 98% of their performance after their least useful 25% of expert neurons were cut.
MoE models send each token to a few experts, which suggests each expert handles a domain like math or code. Other work finds routing mostly tracks token type instead.
They scored each neuron's routed contribution with a measure called RAMP. Neurons picked for one domain overlapped 58% to 74% across samples of that domain, versus 15% to 24% across domains, and sat in mixed experts.
RAMPr prunes by these scores across all experts and kept 91% to 98% of original performance with 75% of neurons left, across 6 models. With 50% left, GPU memory fell 36% to 43% and batched prefill throughput rose 17% to 39%.
If you compress MoE models, prune neurons inside experts by their routed contribution instead of treating each expert as a single unit.
