Self-Distillation

Process where a model extracts knowledge from its own or a teacher's intermediate representations to optimize itself.