To address the numerical mismatch between sampler and trainer that destabilizes LLM reinforcement learning, this work achieves bitwise-zero train-inference consistency on Cloud TPU. Modular kernel patterns and a three-way verification contract align policy evaluation across sampling and training, making the approach easy to adopt in existing multi-host JAX/TPU serving and training pipelines, with support expanding to linear attention and Mixture-of-Experts architectures.
Linear Attention
线性注意力
Mechanism approximating standard attention with linear complexity to cut long-sequence cost.
Curated Intelligence
TokenSpeed introduces a next-generation LLM inference engine optimized for agentic workloads, addressing multi-turn short requests and long-context reuse. On the Blackwell platform, it achieves 11% higher throughput than TensorRT-LLM while significantly improving KV Cache reuse efficiency to reduce costs.
The OpenMOSS team from Fudan University published 'Mechanics of Long-Context Hybrid Models,' systematically investigating hybrid architecture designs for long-context LLMs. The study suggests that combining Full Attention, Sliding Window Attention (SWA), and Gated Linear Attention (GLA/GDN)—a 'take all' strategy—balances computational efficiency and modeling capability better than choosing just one efficient variant. Validated across multiple model scales, this work reveals the synergy between different attention mechanisms, offering theoretical and practical guidance for next-generation efficient long-text models.
This article provides a deep analysis of the evolution of linear attention mechanisms from DeltaNet to KDA, focusing on deriving complex non-linear RNN gating into affine transformations and lower-triangular matrix solving. It reveals the technical logic behind balancing model expressiveness with hardware throughput, ideal for developers seeking to understand LLM inference foundations.
KRAFTON AI has released and open-sourced the preview of Raon-1 mini, a model trained from scratch. Featuring a 30B-A4B Mixture-of-Experts architecture with linear attention and speculative decoding, it is licensed under Apache 2.0. The company claims performance comparable to Meta's Muse Glimmer 30B in coding and agentic tasks, while offering faster inference and better memory efficiency.
Alibaba's Qwen team released FlashQLA, a high-performance GPU operator library designed for Gated Delta Network (GDN) linear attention. By employing hardware-friendly algebraic rewrites and sequence parallelism, it addresses memory redundancy and SFU overload in long sequences, achieving 2-3x speedups in forward and backward passes on Hopper and Blackwell architectures.
i14's Laplace 3B utilizes a linear attention architecture with a fixed recurrent state of 29.5 MB per sequence, independent of context length. By leveraging TorchNeuron Native, the model achieves native PyTorch support on AWS Trainium2 while preserving its Gated Linear Attention (GLA) design and Muon optimizer. Benchmarks show training speeds of ~120k tokens/s and inference throughput of 11.5k output tokens/s across 1,280 concurrent sequences on a single trn2.48xlarge instance.
Subquadratic introduces SubQ Snap, a long-context decision model designed for real enterprise workflows. By leveraging linear attention, it maintains constant compute per token as context scales from 100K to 384K tokens, avoiding the quadratic cost explosion of standard attention mechanisms. The model supports multimodal inputs and outperforms competitor Jev on the Decision Index v0.2.1, bridging the gap between short-context classification tools and full-codebase judgment capabilities.
A team from Beijing University of Posts and Telecommunications introduced Mixture-of-Top-k Attention (MiTA) at NeurIPS'26. By reframing standard attention as a dynamic Feed-Forward Network with width scaling by sequence length via 'Fast Weights', they established a five-dimensional classification framework. MiTA employs deformable, reusable Top-k experts to balance global context retention with on-demand detail retrieval, offering a unified architectural perspective for understanding long-context costs and reconciling diverse attention mechanisms.
Zhipu AI has open-sourced and launched GLM-5.3-Flash (320B-A18B), the first native multimodal model in the GLM-5 series. With an Artificial Analysis Intelligence Index score of 57, it matches Claude Opus 4.8 in coding capabilities while costing only 1/40 as much. By employing a hybrid sparse-linear attention architecture and halving active parameters, GLM-5.3-Flash achieves frontier-level intelligence at significantly lower computational costs.