Hybrid Attention

Hybrid Attention is an architectural design that combines multiple attention mechanisms across different layers of a Transformer network. It applies linear attention with linear compute complexity to most layers while preserving full gated attention on select layers, balancing inference throughput with reasoning expressiveness across long contexts.