Alibaba Qwen Proposes TRACE for 5.4x Faster FP4 RL Rollouts

Using the actual FP4 quantization results from the Rollout side to guide rounding decisions on the training side.

Reinforcement learning (RL) for large models is becoming increasingly "compute-hungry," with the Rollout phase being one of the most expensive components. Compressing Rollout from BF16 to FP4 theoretically offers significant reductions in compute and memory overhead, but this introduces new challenges. Even when numerical values are very close, the discrete quantization boundaries of FP4 can cause the training and Rollout sides to map to different code points. This amplifies minor numerical errors into policy deviations, potentially leading to training collapse.

A research team from Alibaba's Qwen and Ohio State University proposed TRACE. Its core idea is not to further improve the quantization precision of the training and Rollout paths separately, but rather to directly reduce the discrepancy between these two low-precision execution paths: using the actual FP4 quantization results from the Rollout side to guide rounding decisions on the training side. Across four MoE models—Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, Qwen3.8-Flash-Next, and Qwen3.8-2.4T-A95B—TRACE restores final performance to levels comparable to BF16 Rollout under joint FP4 Weight/Activation and FP4 KV Cache Rollout settings. Additionally, it achieves up to 5.4× improvement in decoding throughput for Rollouts at a 128K output length.

• Paper Title: TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
• ArXiv Link: https://arxiv.org/abs/2610.07767
• Huggingface Link: https://huggingface.co/papers/2610.07767

Why Does FP4 Rollout "Break" RL Training?

The appeal of low-precision Rollout is straightforward: the Rollout phase typically consumes a large portion of time in RL, especially as trajectory lengths expand to tens or hundreds of thousands of tokens. Low-precision computation and low-precision KV Cache significantly reduce costs. Compared to FP8, FP4 is more aggressive, offering greater potential speedups, but its issues are also more acute. Existing methods typically focus on "how close the quantized value is to the original high-precision value," i.e., reducing quantization error independently for the training and Rollout paths. The authors of TRACE point out that this does not equate to reducing the difference between the two paths. Two values can each be accurately quantized yet still map to different code points.

Figure 1: Minor differences in BF16 can be amplified into significant discrepancies in low precision after crossing FP4 rounding boundaries.

The paper provides an intuitive example: a training-side activation value of 60.24 and a Rollout-side value of 59.76 differ by only 0.48. After applying the same global scale and block scale, the normalized values become 2.51 and 2.49, respectively. Because they fall on opposite sides of a rounding boundary, one is quantized to 3 and the other to 2, amplifying the difference to 24. For MoE models, such deviations are particularly problematic: numerical changes can affect router scores and expert selection, causing the training policy to diverge further from the policy corresponding to the generated trajectories. In experiments, all baseline methods—including standard FP4 QAT, QaRL, and QUADS—experienced noticeable drops or even collapses in reward and test scores during later stages of training.

Figure 2: On Qwen3.5-35B-A3B, FP4 QAT/QUADS/QaRL show performance degradation, while TRACE remains stable and gradually matches BF16 performance.

TRACE Core Concept:

Don't Just Focus on "Accurate Quantization"—Align Training Directly with Rollout Quantization Results

The first design element of TRACE is Rollout-Guided Quantization-Aware Training. During the Rollout phase, the system records the quantization results corresponding to FP4 routed expert activations and FP4 KV states. In subsequent training phases, this information is no longer just "historical data" but directly participates in rounding decisions for fake quantization on the training side. Specifically, for a value to be quantized on the training side, TRACE still selects between the two adjacent FP4 E2M1 code points but no longer mechanically uses standard round-to-nearest (RTN). Instead, it prioritizes the candidate value closer to the actual code point used on the Rollout side.

Figure 3: Overall TRACE workflow. Compact quantization information saved on the Rollout side is loaded on the training side to adjust FP4 rounding.

However, directly saving the full quantization results from Rollout is not cheap. This information must undergo GPU→CPU transfer, CPU caching, disk write/read operations, and then return to the GPU during training; long trajectories, large batches, and asynchronous RL all amplify storage and communication pressure. The second design aspect of TRACE is precisely minimizing "what needs to be transmitted." The authors found that over 99% of train/rollout quantization inconsistencies differ by only one adjacent FP4 codebook entry. Since the activation values on the training side already narrow down the candidate range significantly, there is no need for the Rollout side to transmit the entire low-precision value; retaining a small amount of mantissa information, along with the scale, is usually sufficient to tell the training side which direction to round.

Experimental Results

On RL tasks using Qwen3.5-35B-A3B, TRACE achieved an average score of 75.3. For comparison, BF16 Rollout scored 74.9, standard QAT scored 59.6, QaRL scored 58.6, and QUADS scored 68.8. Notably, on HMMT25, TRACE improved from QUADS's 59.0 to 70.0.

The paper specifically measured Rollout throughput and end-to-end RL step time. The results show that saving full guidance information indeed severely slows down Rollout due to communication overhead; however, with compact caching, TRACE's throughput closely matches native FP4 Rollout. With 8K input and 128K output, decoding throughput improves by up to 5.4× compared to BF16 Rollout. End-to-end, the additional overhead introduced by TRACE in the RL step time of joint FP4 Rollout is only about 7.4%.

A more interesting phenomenon:

Models "Adapt" to FP4 During RL

Another noteworthy observation in TRACE's experiments is that while FP4 execution causes performance loss in the early stages of training, as long as the train-rollout discrepancy is controlled, the model gradually adapts to this low-precision execution environment.

Figure 4: TRACE's performance adaptation phenomenon. Left: Qwen3.5-35B-A3B; Right: Qwen3.8-Flash-Next.

On the HMMT25 curve for Qwen3.5-35B-A3B, TRACE catches up to BF16 after approximately 120 RL steps; for Qwen3.8-Flash-Next, a similar "catch-up" occurs around 240 steps. This implies that low precision does not necessarily have to be viewed as static noise during training. If training and generation use a consistent, learnable low-precision environment, the policy itself may gradually adapt to these numerical constraints through RL updates. This is also reflected in the quality of the final FP4 policy. If RL is first completed using BF16 Rollout followed by post-training quantization, the average scores for Vanilla NVFP4, 4over6, and H-Scale are 70.4, 71.0, and 71.4, respectively; whereas TRACE uses joint FP4 Rollout throughout the training process, achieving an average score of 75.3 for the final FP4 policy.

© THE END

Please contact this official account for authorization before reprinting.

For submissions or media inquiries: [email protected]