Researchers propose a two-stage recipe for distilling Transformers into Mamba-like architectures: first distill into a linearized Attention model, then transfer to Mamba with a principled initialization, avoiding the performance loss of naive distillation. The work also provides compute-optimal distillation recipes for both existing-teacher and train-from-scratch scenarios, reducing the risks of large-scale distillation.
State Space Model (SSM)
状态空间模型
Sequence modeling architecture with fixed-size state and linear complexity, fast at inference but prone to long-context degradat
Curated Intelligence
A NeurIPS 2026 PALM workshop paper introduces weighted state souping, which learns query-dependent weights to recombine cached state-space model encodings. The method substantially improves long-context performance on single- and multi-hop QA, needle-in-a-haystack retrieval, variable tracking, and tool use, while a sparse variant achieves near-constant encoding cost with respect to context length.
Through theory and experiments, this work shows that the gating mechanism common in modern recurrent networks explains why State Space Models lag behind Transformers on in-context learning and precise retrieval: gating drives SSMs to first learn an in-weights memorization solution, delaying or even preventing convergence to a correct in-context learning solution, even absent architectural or memory-capacity limits. Conversely, gating often benefits generalization to long sequence lengths, revealing its dual role in training dynamics.
AWS Partner Cartesia leverages State Space Models (SSMs) and AWS infrastructure to deliver sub-100ms text-to-speech and speech-to-text at global scale. This solution addresses latency bottlenecks in real-time conversational AI, enabling developers to build voice applications with human-like responsiveness.
The paper proposes a two-stage recipe for distilling Transformers into Mamba-like architectures: first distilling knowledge into a linearized Attention model, then transferring to Mamba with a principled initialization, overcoming the failure of naive distillation to preserve teacher performance. It also provides compute-optimal distillation recipes for both the existing-teacher and jointly-trained-teacher scenarios, reducing risks of large-scale distillation.
Google introduces Titans and MIRAS, proposing a novel sequence model that integrates online optimization, associative memory, and architectural design. This approach addresses the computational bottlenecks of Transformers in ultra-long contexts by incorporating neural long-term memory modules beyond standard Euclidean paradigms, offering superior information capture compared to traditional RNNs/SSMs. The research provides a new theoretical framework and architectural path for building AI systems with true 'long-term memory'.
IBM and the Mamba team released Bamba, an open-source hybrid model combining Transformer and Mamba2 architectures. It achieves up to 5x faster inference than LLaMA on long sequences, significantly optimizing memory usage and computational efficiency.
The Technology Innovation Institute (TII) has released the open-source Falcon 3 family of LLMs, ranging from 1B to 10B parameters. The series emphasizes enhanced capabilities in science, math, and code, notably introducing a 7B base model built on the Mamba architecture to improve training efficiency and inference performance.
TII released Falcon Mamba, the first strong-performing attention-free large language model at the 7B scale. Built on the Mamba architecture, it replaces traditional Transformer attention layers with state space models to significantly reduce inference costs and enhance long-context processing capabilities.