Baidu Open-Sources ERNIE 5.5 Flash Base, a 280B MoE Model Topping Same-Size Benchmarks
Baidu has open-sourced ERNIE 5.5 Flash Base, a Mixture-of-Experts foundation model with 280B total parameters and 14B activated per token, pretrained on 33.5T tokens and supporting up to 1M-token context in fine-tuning and inference. It claims same-size SOTA across knowledge, reasoning, math, code, multilingual and long-context benchmarks, surpassing DeepSeek-V4-Flash-Base and peers. Its efficiency-oriented innovations include Latent-Gated Sparse Attention, Multi-Router Latent MoE and the ECHO training objective that reuses the backbone's final layer.
Full Report
On November 5, Baidu officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base.
On November 5, Baidu officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base. The model adopts an MoE architecture with 280 billion total parameters, activating 14 billion parameters per token. It was pre-trained on 33.5 trillion tokens, with a pre-training context length of up to 128K, and supports a context length of up to 1 million tokens during fine-tuning and inference. It achieves SOTA performance among models of the same size on multiple benchmarks covering knowledge, reasoning, mathematics, code, multilingual capabilities, and long context, surpassing models such as DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base.
Open-source address: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base
As large model applications extend from text Q&A to long-document analysis, multi-turn interaction, and agent tasks, the volume of information models need to process continues to grow, and computational and memory overhead rises accordingly. Improving resource utilization efficiency while ensuring model capability has become an important topic in large model R&D.
ERNE 5.5 Flash Base is built around "efficiency" as its core design principle, systematically optimizing key components such as the attention mechanism, expert routing, and training mechanisms, and achieving multiple algorithmic innovations.
In terms of the attention mechanism, ERNIE proposes Latent-Gated Sparse Attention, which combines hybrid sparse attention, low-rank gated attention, and virtual head attention. Specifically, the model combines HCA and MLA so that different attention layers respectively handle compressing historical information and fine-grained access to historical positions; Low-Rank Gated Attention uses existing low-dimensional representations to compute information gating, reducing gating projection weight parameters by 75%; in addition, Virtual Head Attention (VHA) enhances model expressiveness and training quality, and is fused into the attention operator during inference with minimal overhead. These designs reduce the resource overhead of long-context processing while preserving the model's ability to handle key information.
In terms of expert routing, the model adopts Multi-Router Latent MoE, in which multiple independent routers jointly participate in expert selection, improving the correlation between routing scores and the actual utility of experts. Compared with traditional schemes that rely on a single router, this design aims to reduce the impact of a single scoring bias on expert selection while keeping the number of activated experts, expert computation, and communication volume unchanged. In terms of training objectives, the model introduces ECHO, which requires no additional MTP (Multi-Token Prediction) module; instead, it reuses the last layer of the backbone, allowing the backbone itself to learn to predict subsequent tokens, thereby planning the sequence direction in advance and effectively improving reasoning and code capabilities. ERNIE 5.5 Flash adopts only single-step ECHO pre-training, and the reused last layer can also serve as a draft model for speculative decoding, demonstrating multi-step extrapolation capability: despite training only one step, multi-step drafting quality is substantially better than that of auxiliary MTP modules.
Beyond model architecture, innovation in large model infrastructure is also important. ERNIE optimized the distributed Muon optimizer and introduced FlashMask-4, an attention operator for large head dim, improving communication, memory utilization, and computational efficiency in large-scale training. Improvements in model capability depend not only on the algorithms themselves but also on whether the training system can efficiently support these designs.
Foundation models not only carry the general capabilities formed during pre-training but also serve as an important foundation for downstream model development and application innovation. Unlike application models aimed at end users, Base models provide developers and enterprises with room for further fine-tuning and customization, enabling them to build specialized models and applications suited to different scenarios based on their own business needs. Open foundation models provide developers with more technical choices and lower the training cost and technical barrier for developing specialized models.
Currently, competition in large models is gradually entering a new stage of coordinated development across model capability, training efficiency, and scenario innovation. As technologies and applications such as reasoning and Agents accelerate their evolution, industry demand for foundation models is shifting from single-dimensional capability improvement to more efficient training paradigms, more flexible capability expansion, and broader scenario adaptation needs. While improving the model's foundational capabilities and training/inference efficiency, ERNIE 5.5 Flash Base provides developers and the community with a more extensible technical foundation, driving model technology toward diversified capability exploration and application innovation.
This article is provided by Baidu. QbitAI is authorized to repost it. The views belong to the original author.