Baidu Open-Sources 280B-Parameter ERNIE 5.5 Flash Base, Setting New SOTA for Its Size

Baidu has open-sourced the pre-training endpoint weights of its ERNIE 5.5 Flash Base model. With 280 billion total parameters and a sparse MoE architecture activating only 14 billion parameters per token, the model was pre-trained from scratch on 33.5 trillion tokens. Notably, this release is the base checkpoint without mid-training or post-training, positioned by Baidu as a common starting point for domain adaptation, instruction tuning, and reinforcement learning research; a fully post-trained version will follow.

Technically, the model introduces several efficiency innovations: training context is 128K, but fine-tuning and inference support up to 1 million tokens; training uses a distributed Muon optimizer with FP8 mixed precision and a FlashMask-4 attention operator based on CuTe-DSL, supporting SM90/SM100. It also features Latent-Gated Sparse Attention (combining HCA, MLA, low-rank gated attention, and virtual head attention, reducing gating projection parameters by 75%), Multi-Router Latent MoE (using multiple routers to improve expert selection), and an ECHO training objective that reuses the last layer of the backbone for MTP-free multi-token prediction. The model achieves state-of-the-art results among comparable models on benchmarks covering knowledge, reasoning, math, code, multilingual, and long-context tasks, surpassing models like DeepSeek-V4-Flash-Base.

The open-source release is significant for the AI community: by providing 280B-scale base weights, it lowers the barrier for domain adaptation and reinforcement learning research, and its MoE efficiency innovations offer new directions for architecture evolution, potentially accelerating the industrial adoption of sparse models.