Fudan OpenMOSS Proposes Hybrid Attention Paradigm: Use Both

Why Hybrid Architectures Work, and How to Design Them More Effectively?

The first author of this article is Liu Xiaoran, a combined Master's-PhD student (Class of 2022) at Fudan University and the Shanghai Institute for Advanced Intelligence (Shanghai AI Lab). His advisors are Qiu Xipeng and He Ziwei. His research focuses on long-context large language models (LLMs), and he has published over ten papers in top-tier international AI conferences such as ICLR, ICML, AAAI, and EMNLP.

Background

Long context has become a critical foundational capability for LLMs, but it also presents a practical challenge for model architecture: Full attention can directly model relationships between any positions, but incurs computational and memory overhead that grows significantly with sequence length; Sliding Window Attention (SWA) and Linear Attention are more efficient but suffer from obvious performance deficits. Consequently, hybrid architectures have gradually become the mainstream approach in current LLM design.

Facing two types of efficient solutions with fixed computational and storage complexity—Sliding Window Attention and Linear Attention—which one should hybrid architectures combine with Full Attention? The latest work from Professor Qiu Xipeng’s team at Fudan University’s Shanghai Institute for Advanced Intelligence, titled Mechanics of Long-Context Hybrid Models, offers an interesting answer: "I want them all."

From the perspective of hybrid positioning, the paper systematically compares the performance of hybrid architectures based on RoPE/NoPE Full Attention, Sliding Window Attention (SWA), and Gated Linear Attention (GLA/GDN) in long-context tasks. It further analyzes the synergistic mechanisms between different attention types.

• Paper: https://arxiv.org/abs/2610.10114
• GitHub: https://github.com/OpenMOSS/Hybrid-Mechanics
• Zhihu: https://zhuanlan.zhihu.com/p/2092654065377457038

Observations

The research team conducted experiments across four model scales: 376M, 776M, 1B, and 3B. The models were first pre-trained on short contexts (4k) and then continued training on long contexts (32k). Evaluations covered PG19 PPL, LongPPL, RULER, BABILong, and LongBench, including length extrapolation performance under short-context training and extended-length training performance after long-context continuation. The study also compared Layer-wise Hybridization (LH) and Head-wise Hybridization (HH).

The first observation presented by the paper is that hybrid positioning enables better long-context performance in hybrid architectures. RoPE, SWA, and Gated Linear Attention all incorporate explicit or implicit positional biases, whereas NoPE attention does not introduce positional bias. Mixing NoPE with these position-biased attention mechanisms not only enhances extended-length performance during long-context training but also enables plug-and-play length extrapolation.

Length extrapolation performance of SWA/GLA/GDN-NoPE-LH

Long-context continuation performance of SWA/GLA/GDN-NoPE-LH

However, performance differences still exist among various hybrid architectures. After short-context pre-training, SWA-NoPE hybrids typically demonstrate strong length extrapolation capabilities. Even when trained with a context length of only 4k, the models maintain good performance on longer contexts.

But strong length extrapolation does not imply strong extended-length training. After long-context continuation training, the performance landscape reverses: SWA-NoPE is outperformed by GLA-NoPE or GDN-NoPE, particularly in layer-hybridized models.

The research team refers to this phenomenon as the "seesaw effect" of sliding window hybridization in extended-length training. Correspondingly, while linear hybrid models exhibit stronger fitting capabilities during long-context continuation, they are less proficient at length extrapolation than sliding window hybrid models. This constitutes another core conclusion: there is no free lunch for linear hybrids.

Improving Extended-Length Training for Sliding Window Hybrids

The research team further examined perplexity curves and parameter changes before and after long-context continuation training. For SWA-NoPE layer-hybridized models, loss reduction was significant for short-context positions after long-context continuation, but improvement was limited for long-context positions. During training, the model tended to prioritize learning local information accessible via sliding window attention, failing to fully learn how to utilize distant context.

This is termed the "short-context learning trap": the sliding window branch receives substantial optimization during training, but its fixed window size limits the range of dependencies it can model, ultimately causing long-context training to over-focus on short contexts.

Deeper analysis reveals that short windows facilitate length extrapolation during the short-context pre-training phase; however, during the long-context continuation phase, short windows restrict the model's ability to learn long-context dependencies. The paper summarizes these two trends as "short-window fatigue" and "long-window laziness."

During long-context pretraining, the trend of SWA-NoPE-LH's long-context performance as the window size increases. Larger windows yield better extended training performance, reflecting "short-window fatigue."

During short-context pretraining, the trend of SWA-NoPE-LH's long-context performance as the window size increases. Smaller windows yield better length extrapolation performance, reflecting "long-window laziness."

To address this, the paper significantly improves the long-context continued training performance of SWA-NoPE/RoPE layer-hybrid models by increasing the sliding window size and incorporating the LongCE loss, approaching or even matching the level of linear hybrid models across multiple scales.

Extended training improvement results for SWA-NoPE-LH

Extended training improvement results for SWA-RoPE-LH

Extrapolation Improvements in Linear Hybrids

Linear attention demonstrates strong fitting capabilities during long-context continued training but underperforms compared to sliding-window hybrids when extrapolating beyond the training length. The research team re-examined this issue from the perspective of relative positional information. Although GLA and GDN do not use traditional RoPE, their gating mechanisms inherently contain implicit positional biases. As distance increases, the gating in state updates accumulates continuously, potentially causing distribution shifts beyond the training length.

Based on this hypothesis, the paper proposes Sliding-Window Linear Attention (SWLA): applying sliding-window constraints to the effective relative position range of GLA/GDN during inference, while using logarithmic correction for NoPE attention to enhance its global aggregation capability.

This strategy limits the relative position range perceived by attention modules with positional bias and enhances the global perception of modules without positional bias, achieving excellent extrapolation effects—a phenomenon termed the Matthew effect of hybrid positional encoding. Experiments show that this strategy significantly improves length extrapolation for GLA-NoPE and GDN-NoPE hybrids. On the RULER NIAH-SK1 task, models trained at 4k context length extrapolate to 64k, achieving 16x training-free extrapolation while maintaining 100% accuracy.

Length extrapolation improvement results for GLA-NoPE-LH

Length extrapolation improvement results for GDN-NoPE-LH

How Hybrid Positional Encodings Collaborate

To understand why hybrid architectures are effective, the paper also analyzes the attention characteristics of different attention modules. The results show that NoPE attention tends to perform coarse-grained global aggregation, maintaining high retrieval hit rates in long contexts; whereas RoPE, SWA, and linear attention rely more heavily on positional bias to limit noise and have a greater impact on downstream performance in long-text tasks.

Taking the RoPE-NoPE hybrid as an example, compared to full RoPE attention, full NoPE attention exhibits higher attention score hit rates and larger attention entropy, demonstrating coarse-grained yet effective global aggregation. As the mixing ratio changes, the distribution boundary between NoPE and attention modules with positional bias in the entropy-hit rate space also shifts. The research team terms this collaborative pattern the "tidal effect" of hybrid positional encoding.

Comparison among RoPE-NoPE-HH models with different mixing ratios

Therefore, the key to hybrid architectures lies not merely in placing multiple modules within the same model, but in enabling them to form complementary positional information structures, ultimately achieving performance that surpasses single attention mechanisms.

Conclusion

This work revolves around two seemingly simple questions: Why are hybrid architectures effective, and how can they be designed to be more effective? Six conclusions were drawn:

• Conclusion 1: From Hybrid Attention to Hybrid Positional Encoding (PE): "Hybrid PE" refers to mixing NoPE attention with other position-biased attention mechanisms. This approach proves more effective for length extrapolation and long-context training.
• Conclusion 2: The Seesaw Effect in Sliding Window Hybrid Expansion: Sliding window hybrids (SWA-NoPE) demonstrate superior length extrapolation, but underperform compared to linear hybrids (LA-NoPE) in extended context training, particularly in layer-mixed models.
• Conclusion 3: No Free Lunch for Linear Hybrids: While linear hybrids (LA-NoPE) excel in extended context training, their length extrapolation capabilities are inferior to those of sliding window hybrids (SWA-NoPE).
• Conclusion 4: Tidal Effects of Hybrid PE Synergy: Effective collaboration in hybrid PE relies on a few NoPE attention heads performing coarse-grained aggregation, while the majority of position-biased attention heads limit noise. The distribution boundaries between these two types fluctuate with their ratio in entropy-hit-rate scatter plots.
• Conclusion 5: Short-Window Fatigue vs. Long-Window Laziness: Long-context training with sliding window hybrids faces a "short-context learning trap." Although short windows facilitate length extrapolation during pre-training on shorter texts, longer windows prove beneficial for extended context training during continued pre-training on longer texts.
• Conclusion 6: The Matthew Effect in Hybrid PE Extrapolation: In hybrid PE models, position-biased attention restricts local perception via sliding windows, while NoPE attention enhances aggregation through logarithmic correction, yielding excellent extrapolation performance.

The research team established an analytical chain from phenomenon to mechanism to improvement, starting from differences in long-text performance across training and inference stages and analyzing internal attention behaviors. The results indicate that the advantages of hybrid architectures stem from functional complementarity among different attention modules, rather than any single module being universally superior across all scenarios.

Linear attention is well-suited for efficiently learning and fitting long contexts; sliding window mechanisms are ideal for controlling relative positions to achieve length extrapolation during inference; and NoPE attention provides coarser but effective global aggregation. Organizing these capabilities may represent the future direction for the development of long-context hybrid architectures.

This paper focuses on hybrid architectures combining efficient attention mechanisms—such as sliding window attention and linear attention—with full attention, investigating their performance on long texts during the pre-training stage. In the future, the research team plans to further explore the performance of these hybrid architectures in post-training and multimodal training stages, as well as the performance of sparse attention hybrids and compressed attention hybrids within the same stages.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]

Open in WeChat