Fudan OpenMOSS Proposes Hybrid Attention Paradigm: Use Both

The OpenMOSS team from Fudan University published 'Mechanics of Long-Context Hybrid Models,' systematically investigating hybrid architecture designs for long-context LLMs. The study suggests that combining Full Attention, Sliding Window Attention (SWA), and Gated Linear Attention (GLA/GDN)—a 'take all' strategy—balances computational efficiency and modeling capability better than choosing just one efficient variant. Validated across multiple model scales, this work reveals the synergy between different attention mechanisms, offering theoretical and practical guidance for next-generation efficient long-text models.