Termsslug: gqa

GQA

分组查询注意力

An attention mechanism where multiple query heads share a single set of key-value heads, balancing between MHA and MQA.

Curated Intelligence

6 items · Page 1/1
Latent SpaceT1·Models
68 pts
Meta Releases Muse Spark Open: High-Performance Multimodal MoE Model

Meta released Muse Spark Open, an 816B-parameter Mixture-of-Experts model with 42B active parameters, scoring 39 on the Artificial Analysis Intelligence Index to become the top US open-weight model. It supports multimodal inputs (text, image, video, audio) and uses NVFP4 quantization to reduce weight size to ~530GB, enabling deployment on 4 Blackwell Ultra GPUs or two Mac Studios. While strong in general intelligence benchmarks, it ranks last among top open models in agentic evaluations, highlighting weaknesses in autonomous task execution.

InfoQ 中文T2·Models
86 pts
StepFun Open-Sources Step 5 Preview: 600B MoE Model with Million-Token Context for Long-Horizon Agents

StepFun has released the weights for Step 5 Preview (BF16/FP8), featuring a 600B sparse MoE architecture and million-token context support. Designed for long-horizon agent tasks, it utilizes a 92-layer narrow-deep architecture and Sparse GQA to optimize goal consistency and state management in extended contexts, natively supporting cross-file code modification-execution-verification loops.

IT之家 · 智能时代T2·Models
86 pts
StepFun Releases Step 5 Preview: 600B MoE Model with 1M Context for Long-Horizon Agents

StepFun has open-sourced the weights of Step 5 Preview, a 600B-parameter sparse MoE model (27B active) supporting 1M-token context and multimodal inputs. Key innovations include a 92-layer narrow-deep architecture, Sparse GQA with block-level indexing to reduce computational overhead, and RL strategies optimized for long-horizon agent tasks, positioning it as a top performer on OpenRouter.

vLLM 官方博客T1·Tools
78 pts
vLLM Introduces Decode Context Parallelism to Break MLA and GQA Memory Bottlenecks for Million-Token Agent Workloads

vLLM introduces Decode Context Parallelism (DCP) to address KV cache memory duplication bottlenecks in GQA and MLA architectures during long-context inference. By optimizing context distribution at the decode stage, DCP significantly enhances concurrency and memory efficiency for agent workloads ranging from 64K to 1M tokens on NVIDIA B200 GPUs.

Ahead of AI (Sebastian Raschka)T2·Models
42 pts
Raschka Publishes Visual Guide to LLM Architectures

AI educator Sebastian Raschka launched a visual gallery featuring 45 major LLM architectures, focusing on attention mechanism variants. The resource consolidates past articles and previously undocumented designs into intuitive model cards to aid understanding of architectural principles.