Termsslug: fused-cross-entropy-kernel

Fused Cross-Entropy Kernel

融合交叉熵内核

An implementation that fuses softmax and cross-entropy into a single GPU kernel to save memory and bandwidth.

Curated Intelligence

1 items · Page 1/1
Hacker News AIT2·Models
74 pts
Soft-MTP Fixes Gradient Interference in Multi-Token Prediction

Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.