Prime Intellect Releases Mega MoE Training Kernel, Boosting Speed by 55%

1 reports archived1 independent sourcesupdated 10/30/2026, 02:34:52

Synthesis & Latest Updates

1 Sources Cross-Validated

Prime Intellect released Prime Mega MoE, extending DeepSeek's fused kernel to support efficient backward pass training. On the Qwen3-235B model, this approach reduces single-step training time from 14.98s to 9.65s, achieving a ~55% speedup by optimizing communication and computation overlap in Mixture-of-Experts layers.

LATEST/Prime Intellect released Prime Mega MoE, extending DeepSeek's fused kernel to support efficient backward pass training. On the Qwen3-235B model, this approach reduces single-step training time from 14.98s to 9.65s, achieving a ~55% speedup by optimizing communication and computation overlap in Mixture-of-Experts layers.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

Marktechpost (@Marktechpost)T2Oct 30, 02:34·78 pts
Prime Intellect Releases Mega MoE Training Kernel, Boosting Speed by 55%
- Eliminates GPU idle time in MoE layers by fusing routing, sending, computing, and returning steps.
- Achieves a 55% speedup over TorchTitan baseline when training Qwen3-235B-A22B on an H200 cluster.
- Integrated into prime-rl (Apache 2.0) as the mega_moe dispatch backend, supporting BF16, Hopper FP8, and Blackwell MXFP8.
View Related Reports & Analysis Original Source