Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.
Termsslug: straight-through-gradient-routing
Straight-Through Gradient Routing
直通梯度路由
An engineering technique that substitutes or combines operators in the forward pass while routing gradients along a specified pa
Curated Intelligence
Hacker News AIT2·Models
74 pts