Termsslug: straight-through-gradient-routing

Straight-Through Gradient Routing

直通梯度路由

An engineering technique that substitutes or combines operators in the forward pass while routing gradients along a specified pa

Curated Intelligence

1 items · Page 1/1
Hacker News AIT2·Models
74 pts
Soft-MTP Fixes Gradient Interference in Multi-Token Prediction

Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.