Today for AI
ALL INTELLIGENCE
Baidu Open-Sources 280B-Parameter ERNIE 5.5 Flash Base, Setting New SOTA for Its SizeIT之家 · 智能时代T2·Models

Baidu Open-Sources ERNIE 5.5 Flash Base, a 280B Sparse MoE Model

Original title: 百度文心开源发布 ERNIE 5.5 Flash Base 高效基座模型，总参数 280B
78AI Score

Baidu has open-sourced the pretraining-endpoint weights of ERNIE-5.5-Flash-Base, a sparse MoE language model with 280B total parameters and 14B activated per token, pretrained from scratch on 33.5T tokens with up to 1M-token context for fine-tuning and inference. The weights are released without mid- or post-training, positioned as a shared starting point for domain adaptation, instruction tuning and RL research, with the post-trained ERNIE 5.5 Flash to follow.

Full Report

IT之家 November 5 news, Baidu's ERNIE large model today released ERNIE-5.5-Flash-Base, open-sourcing the weights at the pre-training endpoint of ERNIE 5.5 Flash.

This is a sparse MoE language model with 280B total parameters, 14B activated per token, pre-trained from scratch on 33.5T tokens; the maximum training context is 128K, and fine-tuning and inference support a 1M-token context.

The released weights have not undergone mid-training or post-training, and can serve as a common starting point for domain adaptation, instruction fine-tuning, and reinforcement learning research.

The official statement notes that the Base model is only the starting point. The post-trained ERNIE 5.5 Flash will be released subsequently.

Baidu ERNIE open-sources and releases ERNIE 5.5 Flash Base, an efficient foundation model with 280B total parameters

ERNIE-5.5-Flash-Base completed pre-training from scratch on 33.5T tokens, using the Muon optimizer and FP8 mixed precision. The pre-training corpus covers a wide range of domains and languages, with focused development in three directions:

  • Web data: multilingual web crawling, high-fidelity text extraction, and model-driven multi-dimensional quality filtering.
  • Code data: repository and web code underwent non-code filtering, deduplication, and quality grading, supplemented with software evolution and execution-verification data such as commits, issues, pull requests, code reviews, automatically generated tests, and CI-verified fixes.
  • Long-document data: curated natural long documents, long sequences mixed with semantically similar retrieval distractor text, and synthetic retrieval, citation-based question answering, and multi-hop question answering data.

Baidu ERNIE open-sources and releases ERNIE 5.5 Flash Base, an efficient foundation model with 280B total parameters

▲ Model architecture

According to the introduction, the training system and model architecture were co-designed. The optimizer uses distributed Muon: at the scale of thousands of GPUs, cross-machine communication for Dense parameters and MoE parameters is merged into a single collective communication, reducing cross-machine latency and jitter. The attention operator uses FlashMask-4 based on CuTe-DSL, supports SM90 and SM100 architectures, supports head dim up to 576 (qk) / 512 (v), and can be directly used for computation of 512-dimensional wide heads in HCA layers.

IT之家 attaches the evaluation data and open-source links:

The ERNIE large model evaluated ERNIE-5.5-Flash-Base on 30 benchmarks, covering seven categories of capabilities: general, knowledge, science, reasoning, multilingual, code, and long context, and lists the results of same-scale Base models DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base. All four models are based on their respective publicly released checkpoints and use the same evaluation framework. ERNIE-5.5-Flash-Base performs well across all capability dimensions; in terms of long context, although 37 of the 43 layers perform attention on sequences compressed 128 times, 6 layers of MLA-DSA are sufficient to support fine-grained retrieval. The complete table is available in the Hugging Face model card.

Baidu ERNIE open-sources and releases ERNIE 5.5 Flash Base, an efficient foundation model with 280B total parameters

Advertising statement: External redirect links contained in the article (including but not limited to hyperlinks, QR codes, passwords, and other forms) are used to convey more information and save selection time. The results are for reference only. All articles on IT之家 containing external links include this statement.