Meta released Muse Spark Open, an 816B-parameter Mixture-of-Experts model with 42B active parameters, scoring 39 on the Artificial Analysis Intelligence Index to become the top US open-weight model. It supports multimodal inputs (text, image, video, audio) and uses NVFP4 quantization to reduce weight size to ~530GB, enabling deployment on 4 Blackwell Ultra GPUs or two Mac Studios. While strong in general intelligence benchmarks, it ranks last among top open models in agentic evaluations, highlighting weaknesses in autonomous task execution.
GQA
分组查询注意力
An attention mechanism where multiple query heads share a single set of key-value heads, balancing between MHA and MQA.
Curated Intelligence
StepFun has released the weights for Step 5 Preview (BF16/FP8), featuring a 600B sparse MoE architecture and million-token context support. Designed for long-horizon agent tasks, it utilizes a 92-layer narrow-deep architecture and Sparse GQA to optimize goal consistency and state management in extended contexts, natively supporting cross-file code modification-execution-verification loops.
StepFun has open-sourced the weights of Step 5 Preview, a 600B-parameter sparse MoE model (27B active) supporting 1M-token context and multimodal inputs. Key innovations include a 92-layer narrow-deep architecture, Sparse GQA with block-level indexing to reduce computational overhead, and RL strategies optimized for long-horizon agent tasks, positioning it as a top performer on OpenRouter.
vLLM introduces Decode Context Parallelism (DCP) to address KV cache memory duplication bottlenecks in GQA and MLA architectures during long-context inference. By optimizing context distribution at the decode stage, DCP significantly enhances concurrency and memory efficiency for agent workloads ranging from 64K to 1M tokens on NVIDIA B200 GPUs.
AI educator Sebastian Raschka launched a visual gallery featuring 45 major LLM architectures, focusing on attention mechanism variants. The resource consolidates past articles and previously undocumented designs into intuitive model cards to aid understanding of architectural principles.
Microsoft releases Differential Transformer V2, optimizing the differential attention mechanism by subtracting two heads within the same GQA group that share keys and values. This update aims to improve computational performance and simplify implementation, with code now open-sourced.