REINFORCE Zero Achieves Zero-Kernel-Mismatch Async RL Training

1 reports archived1 independent sourcesupdated 11/6/2026, 08:00:00

Synthesis & Latest Updates

1 Sources Cross-Validated

The work decomposes the rollout-to-learner gap in asynchronous RL into smooth weight lag, which unclipped importance sampling can correct, and fluctuating kernel mismatch, which it cannot. Aligning kernels enables stable training for 20,000 updates at policy lag 64 and 2.40x the throughput of synchronous training, with support for DeepSeek-V4.1-Flash, Kimi K3.1, and AMD MI355X.

LATEST/The work decomposes the rollout-to-learner gap in asynchronous RL into smooth weight lag, which unclipped importance sampling can correct, and fluctuating kernel mismatch, which it cannot. Aligning kernels enables stable training for 20,000 updates at policy lag 64 and 2.40x the throughput of synchronous training, with support for DeepSeek-V4.1-Flash, Kimi K3.1, and AMD MI355X.

Hourly heat curve

3 fully observed hours

Now

12.9

Peak

13.501:00

24h change

–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first

vLLM 官方博客T1Nov 6, 08:00·78 pts
REINFORCE Zero Achieves Zero-Kernel-Mismatch Async RL Training
- Async RL error splits into smooth weight lag and fluctuating kernel mismatch, the latter uncorrectable by importance sampling
- With aligned kernels, unclipped importance sampling trains stably for 20,000 updates at policy lag 64
- Reaches 2.40x synchronous throughput, with support for DeepSeek-V4.1-Flash, Kimi K3.1, and AMD MI355X
View Related Reports & Analysis Original Source