Moore Threads Open-Sources Megatron-MUSA-Titan for Trillion-Parameter MoE Training
Moore Threads has open-sourced Megatron-MUSA-Titan, an infrastructure project optimized for training trillion-parameter MoE models on 10k-GPU clusters. It integrates blockwise FP8, DeepEP, and async communication to enhance Model FLOPs Utilization (MFU) and stability across pretraining and RL scenarios.
Full Report
IT Home reported on October 28 that Moore Threads has announced the open-sourcing of its large model training infrastructure project, Megatron-MUSA-Titan. Built on Moore Threads' full-featured GPUs, the project focuses on the training requirements for trillion-parameter MoE models in ten-thousand-card clusters and covers both pre-training and reinforcement learning scenarios, providing developers with training adaptation and system optimization capabilities for the MUSA platform.
According to the introduction, Megatron-MUSA-Titan has been continuously applied in the pre-training of trillion-parameter MoE models on Moore Threads' ten-thousand-card clusters. Through extensive long-term training practices, it achieves high MFU (Model FLOPs Utilization) and stable long-duration training. It also supports integration with the slime and miles reinforcement learning frameworks, combining Megatron training with SGLang sample generation to build a unified training foundation spanning trillion-parameter MoE pre-training and reinforcement learning.
IT Home noted that Megatron-MUSA-Titan supports blockwise FP8 training, introducing the low-precision computing capabilities of Moore Threads' full-featured GPUs into the training workflow to enhance computational efficiency for large-scale training. Additionally, by leveraging DeepEP and the ACE asynchronous communication engine, it accelerates token dispatching and result aggregation in MoE architectures. This enables overlap between expert parallelism (EP) communication and computation, allowing data exchange and compute tasks to execute concurrently, thereby improving end-to-end training efficiency.
Reference:
https://github.com/MooreThreads/Megatron-MUSA-Titan
Advertisement Disclaimer: External links contained in this article (including but not limited to hyperlinks, QR codes, passwords, etc.) are provided to convey additional information and save selection time. Results are for reference only. All IT Home articles containing external links include this disclaimer.