New Two-Stage Recipe Boosts Transformer-to-Mamba Distillation

1 reports archived1 independent sourcesupdated 11/11/2026, 21:29:46

Synthesis & Latest Updates

1 Sources Cross-Validated

Researchers propose a two-stage recipe for distilling Transformers into Mamba-like architectures: first distill into a linearized Attention model, then transfer to Mamba with a principled initialization, avoiding the performance loss of naive distillation. The work also provides compute-optimal distillation recipes for both existing-teacher and train-from-scratch scenarios, reducing the risks of large-scale distillation.

LATEST/Researchers propose a two-stage recipe for distilling Transformers into Mamba-like architectures: first distill into a linearized Attention model, then transfer to Mamba with a principled initialization, avoiding the performance loss of naive distillation. The work also provides compute-optimal distillation recipes for both existing-teacher and train-from-scratch scenarios, reducing the risks of large-scale distillation.

Hourly heat curve

8 fully observed hours

Now

11.4

Peak

13.511-12

24h change

–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first

Hacker News AIT2Nov 11, 21:29·71 pts
New Two-Stage Recipe Boosts Transformer-to-Mamba Distillation
- Naive Transformer-to-Mamba distillation loses teacher performance and needs principled initialization
- A two-stage pipeline distilling into linearized Attention before Mamba outperforms direct distillation
- Compute-optimal recipes are provided for both existing-teacher and train-from-scratch settings
View Related Reports & Analysis Original Source