Termsslug: compressed-attention

Compressed Attention

压缩注意力

Methods reducing computational load by lowering attention matrix dimensions or sparsification.

Curated Intelligence

2 items · Page 1/1
Marktechpost (@Marktechpost)T2·Models
78 pts
Baidu Releases ERNIE 5.5 Flash Base, a 280B Sparse MoE Model

Baidu released ERNIE 5.5 Flash Base, a 280B-total, 14B-active sparse MoE model shipped directly from pre-training with no mid- or post-training. Efficiency is the design focus: 37 of 43 layers use Heavily Compressed Attention caching one entry per 128 tokens, multiple routers select top-10 of 512 experts, and the final layer doubles as a speculative-decoding draft model. Baidu self-reports gains over DeepSeek-V4-Flash-Base on MMLU-Pro and HumanEval+, but trails on HELMET-Rerank, and all numbers are self-reported.

Ahead of AI (Sebastian Raschka)T2·Models
68 pts
LLM Architecture Trends: Optimizing for Long-Context Efficiency

Recent open-weight LLM releases (e.g., Gemma 4, DeepSeek V4) show a clear architectural shift toward long-context efficiency. Developers are employing techniques like KV sharing, compressed attention, and per-layer embeddings to mitigate memory and compute bottlenecks inherent in reasoning models and agent workflows.