Termsslug: kl-divergence

KL Divergence

KL 散度

Information-theoretic measure of difference between two distributions

Curated Intelligence

5 items · Page 1/1
Hacker News AIT2·Models
74 pts
Soft-MTP Fixes Gradient Interference in Multi-Token Prediction

Researchers identify a flaw in standard Multi-Token Prediction (MTP) training: the MTP cross-entropy gradient contains a low signal-to-noise residual that increasingly interferes with both next-token and multi-token learning at scale. They introduce a KL gradient-recovery trick that exactly recovers the KL self-distillation term from cross-entropy gradients via straight-through gradient routing, yielding Soft-MTP—a one-line code change requiring no vocabulary-sized logit matrices and compatible with fused cross-entropy kernels. At 1.2B scale it improves math and code reasoning after fine-tuning and matches or exceeds direct KL minimization on most metrics.

vLLM 官方博客T1·Models
78 pts
vLLM Achieves Bitwise-Consistent RL Training: Eliminating Numerical Drift to Zero KL Divergence on DeepSeek-V4-Flash at 91% Throughput

To address policy mismatch in RL caused by inference-side optimizations, vLLM engineers modified inference kernels to ensure bitwise-identical logprobs between training and rollout (Diff=0). This approach reduced train-inference KL divergence to zero on DeepSeek-V4-Flash while maintaining 91.3% of baseline throughput, and prevented significant score drops on Nemotron's hybrid architecture. The workflow is now released as an open-source agent skill.

Hugging Face BlogT1·Models
68 pts
Hugging Face Open-Sources Efficient Knowledge Distillation Implementation

Hugging Face and Multiverse Computing have released and open-sourced a new Chunked KL Loss implementation for knowledge distillation. This method significantly reduces peak VRAM usage during training by computing losses in chunks, making large-scale model distillation feasible on standard hardware. The paper is published, and the code is available on GitHub.

Hugging Face BlogT1·Tools
68 pts
vLLM V1 Migration: Fix RL Correctness Before Corrections

The ServiceNow AI team discovered that logprob discrepancies between vLLM V0 and V1 disrupted training dynamics in their PipelineRL framework. By fixing four underlying issues including rollout logprob processing and weight update paths, they achieved alignment with the V0 reference. The key takeaway is to prioritize backend correctness before implementing complex mismatch correction mechanisms.

OpenAI 官方博客T1·Models
78 pts
OpenAI Reveals Scaling Laws for Reward Model Overoptimization

OpenAI researchers identified a significant divergence between true human preferences and proxy reward model scores as policy optimization deepens in RLHF. The study quantifies scaling laws for this 'overoptimization' phenomenon, highlighting KL divergence constraints as a key mitigation strategy, providing critical theoretical grounding for reward modeling in AI alignment.