Efficient sequence modeling architecture based on selective state space, with high generation throughput and low memory use.
Researchers propose a two-stage recipe for distilling Transformers into Mamba-like architectures: first distill into a linearized Attention model, then transfer to Mamba with a principled initialization, avoiding the performance loss of naive distillation. The work also provides compute-optimal distillation recipes for both existing-teacher and train-from-scratch scenarios, reducing the risks of large-scale distillation.
The paper proposes a two-stage recipe for distilling Transformers into Mamba-like architectures: first distilling knowledge into a linearized Attention model, then transferring to Mamba with a principled initialization, overcoming the failure of naive distillation to preserve teacher performance. It also provides compute-optimal distillation recipes for both the existing-teacher and jointly-trained-teacher scenarios, reducing risks of large-scale distillation.