Termsslug: sparse-attention

Sparse Attention

稀疏注意力

Mechanism letting each query attend to only a few relevant historical tokens to cut long-context compute.

Curated Intelligence

23 items · Page 1/3
机器之心 (微信公众号)T2·Models
78 pts
Prefrontal AI and ETH Zurich Unveil LSA to Speed Ultra-Long Context Attention

London startup Prefrontal AI and ETH Zurich jointly propose Latent Sparse Attention (LSA), which targets the indexer overhead that dominates sparse attention at extreme context lengths. At 10M tokens it speeds up single-layer prefill by 22.7x on GLM-5.3-Flash and 72.4x on Qwen3.8-27B while keeping accuracy close to the original models, offering a subquadratic path toward 10M-token context.

Marktechpost (@Marktechpost)T2·Models
88 pts
Alibaba Releases Qwen4-Max 2.4T-Parameter MoE Flagship

Alibaba released Qwen4-Max, a 2.4T-parameter sparse MoE flagship that activates only 71B parameters per token at $1.5 per million input tokens. Its rebuilt sparse MoE stack uses three Gated DeltaNet layers to compress history for each Qwen Sparse Attention layer, a lightweight indexer for context selection, plus four gated residual branches and 328B N-gram embedding parameters. It tops 4 of 16 benchmarks but trails Claude Opus 5.5 and Kimi K3.1 on coding and terminal tasks, a clear trade-off.

IT之家 · 智能时代T2·Models
88 pts
Alibaba Launches 2.4T-Parameter Qwen4-Max, Cutting Task Costs Over 60%

Alibaba's Qwen team has officially launched Qwen4-Max, a 2.4-trillion-parameter sparse MoE flagship with native vision understanding and a 1M-token context window, claiming over 60% lower task cost at comparable performance. The model introduces systematic architectural changes including Gated DeltaNet, Qwen Sparse Attention, Gated Residual and N-gram Embedding, targeting long-horizon professional work in office, coding and research, and is now available via QwenCloud API with smaller open-source variants to follow.

IT之家 · 智能时代T2·Infra
58 pts
Moore Threads Completes Day-0 Adaptation of Kimi K3.1 Pro for Text and Image Inference

Moore Threads announced Day-0 adaptation of Moonshot's open-weight Kimi K3.1 Pro on its MTT S5000 AI card and MUSA software stack, covering model loading, service startup, and text/image inference. Key work includes adapting sparse attention DSA indexing and KV cache layout, online FP8 conversion of MXFP4 expert weights, and integrating visual attention. It signals faster domestic GPU software-stack readiness for trillion-parameter MoE models.

量子位官网T2·Models
78 pts
Baidu Open-Sources ERNIE 5.5 Flash Base, a 280B MoE Model Topping Same-Size Benchmarks

Baidu has open-sourced ERNIE 5.5 Flash Base, a Mixture-of-Experts foundation model with 280B total parameters and 14B activated per token, pretrained on 33.5T tokens and supporting up to 1M-token context in fine-tuning and inference. It claims same-size SOTA across knowledge, reasoning, math, code, multilingual and long-context benchmarks, surpassing DeepSeek-V4-Flash-Base and peers. Its efficiency-oriented innovations include Latent-Gated Sparse Attention, Multi-Router Latent MoE and the ECHO training objective that reuses the backbone's final layer.

Marktechpost (@Marktechpost)T2·Models
78 pts
Zyphra Introduces ZHA Hierarchical Attention for Efficient Long-Context Inference

Zyphra introduces ZHA (Hierarchical Attention), a sparse attention method utilizing a tree structure over the KV cache for efficient retrieval. On Qwen3.6-35B, it outperforms dense attention on LongBench v2 while reading only ~1.1% of keys, with minimal training overhead (optimizing just 80k parameters). Although trade-offs exist in specific benchmarks like Terminal-Bench, it offers a novel architectural approach to long-context processing.

量子位官网T2·Models
78 pts
NVIDIA and MIT Open-Source H3 Video Enhancement with Native 4K Regeneration

NVIDIA and MIT researchers released Open-H3-Regenerate-2K on Hugging Face, a video enhancement project based on MiniMax H3. Using LoRA fine-tuning and latent space amplification, the solution regenerates low-res drafts into up to 4K resolution, achieving ~3.9x faster inference than unoptimized versions and outperforming the official API in perceptual quality benchmarks.

vLLM 官方博客T1·Tools
68 pts
vLLM Delivers Day-0 Support for MiniMax-M3.1-Flash

vLLM announces Day-0 support for the MiniMax-M3.1-Flash model on NVIDIA Blackwell/Rubin and AMD MI350 GPUs. The integration features NVFP4 KV caching, optimized sparse attention kernels, and speculative decoding, significantly improving inference efficiency and throughput for long contexts.