Cross-Architecture Distillation

The paper proposes a two-stage recipe for distilling Transformers into Mamba-like architectures: first distilling knowledge into a linearized Attention model, then transferring to Mamba with a principled initialization, overcoming the failure of naive distillation to preserve teacher performance. It also provides compute-optimal distillation recipes for both the existing-teacher and jointly-trained-teacher scenarios, reducing risks of large-scale distillation.