Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.
Termsslug: fused-cross-entropy-kernel
Fused Cross-Entropy Kernel
融合交叉熵内核
An implementation that fuses softmax and cross-entropy into a single GPU kernel to save memory and bandwidth.
Curated Intelligence
Hacker News AIT2·Models
74 pts