Baidu released ERNIE 5.5 Flash Base, a 280B-total, 14B-active sparse MoE model shipped directly from pre-training with no mid- or post-training. Efficiency is the design focus: 37 of 43 layers use Heavily Compressed Attention caching one entry per 128 tokens, multiple routers select top-10 of 512 experts, and the final layer doubles as a speculative-decoding draft model. Baidu self-reports gains over DeepSeek-V4-Flash-Base on MMLU-Pro and HumanEval+, but trails on HELMET-Rerank, and all numbers are self-reported.
Termsslug: compressed-attention
Compressed Attention
压缩注意力
Methods reducing computational load by lowering attention matrix dimensions or sparsification.
Curated Intelligence
Marktechpost (@Marktechpost)T2·Models
78 pts
Ahead of AI (Sebastian Raschka)T2·Models
68 pts
Recent open-weight LLM releases (e.g., Gemma 4, DeepSeek V4) show a clear architectural shift toward long-context efficiency. Developers are employing techniques like KV sharing, compressed attention, and per-layer embeddings to mitigate memory and compute bottlenecks inherent in reasoning models and agent workflows.