Termsslug: linear-attention

Linear Attention

线性注意力

Mechanism approximating standard attention with linear complexity to cut long-sequence cost.

Curated Intelligence

14 items · Page 1/2
Hacker News AIT2·Models
74 pts
Bitwise-Zero Train-Inference Mismatch Enables Stable On-Policy RL on TPU

To address the numerical mismatch between sampler and trainer that destabilizes LLM reinforcement learning, this work achieves bitwise-zero train-inference consistency on Cloud TPU. Modular kernel patterns and a three-way verification contract align policy evaluation across sampling and training, making the approach easy to adopt in existing multi-host JAX/TPU serving and training pipelines, with support expanding to linear attention and Mixture-of-Experts architectures.

InfoQ 中文T2·Infra
68 pts
TokenSpeed Unveils Next-Gen LLM Inference Engine for Agents

TokenSpeed introduces a next-generation LLM inference engine optimized for agentic workloads, addressing multi-turn short requests and long-context reuse. On the Blackwell platform, it achieves 11% higher throughput than TensorRT-LLM while significantly improving KV Cache reuse efficiency to reduce costs.

机器之心 (微信公众号)T2·Models
78 pts
Fudan OpenMOSS Proposes Hybrid Attention Paradigm: Use Both

The OpenMOSS team from Fudan University published 'Mechanics of Long-Context Hybrid Models,' systematically investigating hybrid architecture designs for long-context LLMs. The study suggests that combining Full Attention, Sliding Window Attention (SWA), and Gated Linear Attention (GLA/GDN)—a 'take all' strategy—balances computational efficiency and modeling capability better than choosing just one efficient variant. Validated across multiple model scales, this work reveals the synergy between different attention mechanisms, offering theoretical and practical guidance for next-generation efficient long-text models.

腾讯技术工程 (微信公众号)T2·Models
78 pts
Deep Dive into Linear Attention Evolution and AI Infra Implementation

This article provides a deep analysis of the evolution of linear attention mechanisms from DeltaNet to KDA, focusing on deriving complex non-linear RNN gating into affine transformations and lower-triangular matrix solving. It reveals the technical logic behind balancing model expressiveness with hardware throughput, ideal for developers seeking to understand LLM inference foundations.

IT之家 · 智能时代T2·Models
78 pts
KRAFTON AI Open-Sources Raon-1 Mini: A 30B MoE Model Rivaling Meta's Muse Glimmer

KRAFTON AI has released and open-sourced the preview of Raon-1 mini, a model trained from scratch. Featuring a 30B-A4B Mixture-of-Experts architecture with linear attention and speculative decoding, it is licensed under Apache 2.0. The company claims performance comparable to Meta's Muse Glimmer 30B in coding and agentic tasks, while offering faster inference and better memory efficiency.

InfoQ 中文T2·Infra
78 pts
Qwen Team Releases FlashQLA for High-Performance Linear Attention

Alibaba's Qwen team released FlashQLA, a high-performance GPU operator library designed for Gated Delta Network (GDN) linear attention. By employing hardware-friendly algebraic rewrites and sequence parallelism, it addresses memory redundancy and SFU overload in long sequences, achieving 2-3x speedups in forward and backward passes on Hopper and Blackwell architectures.

AWS Machine Learning BlogT1·Infra
68 pts
i14's Laplace 3B Linear Attention Model Runs on AWS Trainium2 with Native PyTorch

i14's Laplace 3B utilizes a linear attention architecture with a fixed recurrent state of 29.5 MB per sequence, independent of context length. By leveraging TorchNeuron Native, the model achieves native PyTorch support on AWS Trainium2 while preserving its Gated Linear Attention (GLA) design and Muon optimizer. Benchmarks show training speeds of ~120k tokens/s and inference throughput of 11.5k output tokens/s across 1,280 concurrent sequences on a single trn2.48xlarge instance.

Rohan Paul (@rohanpaul_ai)T2·Models
68 pts
SubQ Snap Launches Long-Context Decision Model with Linear Attention for Constant Compute at 384K Tokens

Subquadratic introduces SubQ Snap, a long-context decision model designed for real enterprise workflows. By leveraging linear attention, it maintains constant compute per token as context scales from 100K to 384K tokens, avoiding the quadratic cost explosion of standard attention mechanisms. The model supports multimodal inputs and outperforms competitor Jev on the Decision Index v0.2.1, bridging the gap between short-context classification tools and full-codebase judgment capabilities.

新智元 (微信公众号)T2·Models
68 pts
BUPT Proposes MiTA: Reinterpreting Attention as Fast-Weight FFN to Unify Linear and MoE Techniques

A team from Beijing University of Posts and Telecommunications introduced Mixture-of-Top-k Attention (MiTA) at NeurIPS'26. By reframing standard attention as a dynamic Feed-Forward Network with width scaling by sequence length via 'Fast Weights', they established a five-dimensional classification framework. MiTA employs deformable, reusable Top-k experts to balance global context retention with on-demand detail retrieval, offering a unified architectural perspective for understanding long-context costs and reconciling diverse attention mechanisms.

智谱 AI 研究T1·Models
88 pts
Zhipu Open-Sources GLM-5.3-Flash: Multimodal Model Matches Claude Opus at 1/40 Cost

Zhipu AI has open-sourced and launched GLM-5.3-Flash (320B-A18B), the first native multimodal model in the GLM-5 series. With an Artificial Analysis Intelligence Index score of 57, it matches Claude Opus 4.8 in coding capabilities while costing only 1/40 as much. By employing a hybrid sparse-linear attention architecture and halving active parameters, GLM-5.3-Flash achieves frontier-level intelligence at significantly lower computational costs.