Tencent Hunyuan Open-Sources GradLoc: Pinpointing RLVR Gradient Spikes and Stabilizing Training via Layerwise Clipping
By Guanhua Huang;Tingqiang Xu;Jinbo WangOriginal title: Stabilizing RLVR via Token-level Gradient Diagnosis and Layerwise Clipping
78AI Score
Addressing frequent training collapse in RLVR caused by gradient spikes, Tencent Hunyuan introduces a white-box diagnostic paradigm. The team released GradLoc, a distributed binary-search tool that pinpoints the single culprit token responsible for anomalies, alongside LayerClip, which enforces constraints based on local layer statistics to prevent suppressing healthy layers during global clipping, significantly extending stable training horizons.
Related Entities
Full Report
GradLoc Code Repository: Tencent-Hunyuan/GradLoc
!12 Figure 1: From black-box heuristics to white-box diagnostics for RLVR training collapse. ! # Key Takeaways 🔥 Key Takeaways
- RLVR Training Collapse with Gradient Spikes: The scalability of RLVR in large language models is hindered by frequent training collapse with gradient spikes. Prior mitigation methods are largely black-box and heuristic: they can alleviate training-inference mismatch, but often fail to deliver long-horizon stability because gradient spikes have multiple causes.
- Localizing the Gradient Spike to a Single Culprit Token: Moving from black-box heuristics to white-box diagnosis, our distributed binary-search Gradient Anomaly Localizer (GradLoc) traces gradient spikes back to the exact culprit tokens. It achieves this with complexity and high success rates. (see §GradLoc)
- New Collapse Mode: Layerwise Gradient Heterogeneity: GradLoc reveals that anomalies stem from semantically coherent tokens with pathological numerical properties (e.g., extreme importance sampling (IS) ratios), rather than "dirty data". Beyond the known training-inference inconsistency, GradLoc identifies a new anomaly class: Tokens maintain safe IS ratios but exhibit severe layerwise gradient heterogeneity, where specific layers explode while others remain stable. (see §Type B (New): Layerwise Gradient Heterogeneity)
- Layerwise Gradient Clipping (LayerClip): To address this heterogeneity, we propose LayerClip, which enforces constraints based on local layer statistics to avoid suppressing healthy layers in global gradient clipping. This approach significantly extends stable training. (see §Addressing Type B: Layerwise Gradient Clipping)