Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.
KL Divergence
KL 散度
Information-theoretic measure of difference between two distributions
Curated Intelligence
To address policy mismatch in RL caused by inference-side optimizations, vLLM engineers modified inference kernels to ensure bitwise-identical logprobs between training and rollout (Diff=0). This approach reduced train-inference KL divergence to zero on DeepSeek-V4-Flash while maintaining 91.3% of baseline throughput, and prevented significant score drops on Nemotron's hybrid architecture. The workflow is now released as an open-source agent skill.
Hugging Face and Multiverse Computing have released and open-sourced a new Chunked KL Loss implementation for knowledge distillation. This method significantly reduces peak VRAM usage during training by computing losses in chunks, making large-scale model distillation feasible on standard hardware. The paper is published, and the code is available on GitHub.
The ServiceNow AI team discovered that logprob discrepancies between vLLM V0 and V1 disrupted training dynamics in their PipelineRL framework. By fixing four underlying issues including rollout logprob processing and weight update paths, they achieved alignment with the V0 reference. The key takeaway is to prioritize backend correctness before implementing complex mismatch correction mechanisms.
OpenAI researchers identified a significant divergence between true human preferences and proxy reward model scores as policy optimization deepens in RLHF. The study quantifies scaling laws for this 'overoptimization' phenomenon, highlighting KL divergence constraints as a key mitigation strategy, providing critical theoretical grounding for reward modeling in AI alignment.