PSN-RLVR Enhances LLM Reasoning Diversity via Parameter Perturbation
Researchers from Shanghai Institute of AI and others proposed PSN-RLVR, which applies consistent small random perturbations to model parameters throughout the reasoning trajectory. This method addresses the issue where RL-trained models become accurate but lose diversity in solution paths, significantly improving Pass@K success rates and out-of-distribution generalization. The paper has been accepted by NeurIPS 2026.
Full Report
Contents7 sections
Giving Large Models a New "Parameter Brain"
Submitted by the PSN-RLVR Team
QbitAI | WeChat Official Account: QbitAI
Models like DeepSeek-R1 have demonstrated the potential of Reinforcement Learning (RL). However, existing research suggests that higher accuracy does not necessarily mean the model can solve more unique problems.
How can we encourage large models to break out of familiar patterns and explore new problem-solving strategies?
A research team from Shanghai Institute for Advanced Study, Fudan University, Shanghai AI Lab, and The Chinese University of Hong Kong proposed PSN-RLVR. This method introduces small perturbations to the model's parameters before solving a problem and maintains these perturbations throughout the entire reasoning process—effectively giving the model a temporary "parameter brain."
Experiments in the paper show that this method improves success rates across multiple attempts on various models. It also achieves out-of-distribution (OOD) generalization benefits in scientific question answering and programming tasks, uncovering solution paths that were absent in baseline multi-sampling results.
The related paper has been accepted to NeurIPS 2026.
From "Higher Accuracy" to "New Solutions"
As models trained with reinforcement learning achieve better scores, a deeper question arises: Has the model mastered more diverse problem-solving methods, or is it simply becoming more proficient at applying methods it already knew? A study covering various public models and RLVR algorithms found that in their test settings, RL-trained models achieved higher single-attempt accuracy. However, when both the original and RL-trained models were allowed many attempts, the original model actually solved more distinct problems.
This explains why it is crucial to distinguish between two metrics: pass@1 measures the success rate of a single attempt, while pass@256 measures the probability of getting at least one correct answer after 256 attempts. Being more consistent in single answers and finding more solutions through repeated attempts are two separate challenges that need to be evaluated independently. Consequently, exploration becomes a key issue: How can we give the model more opportunities to discover different reasoning paths during iterative training?
To enhance exploration, previous studies have attempted:
- Adjusting sampling temperature to increase the model's choice diversity;
- Modifying training objectives to encourage retaining diverse answers;
- Supplementing problems and solution demonstrations to expose the model to more approaches.
However, gradually increasing randomness may not lead to coherent new solutions, and adding learning materials incurs additional costs.
The Shanghai Institute for Advanced Study, in collaboration with Fudan University and other institutions, proposed PSN-RLVR and implemented PSN-GRPO based on GRPO. This approach shifts exploration to the model's internal parameters: Introducing small perturbations before solving and maintaining them throughout the entire reasoning trajectory. This can be understood as temporarily switching to a different "parameter brain," allowing the model to complete the task with a different thinking tendency.
△ Figure 1 from the original paper. The left side compares performance across different numbers of attempts; the right side examines the differences in meaning and computational steps of the responses. Compared to standard GRPO, PSN-GRPO restores some diversity and improves success rates over multiple attempts.
How PSN Differs: Maintaining Perturbations Throughout Reasoning
Model parameters can be viewed as a set of internal settings that influence how the model thinks and expresses itself. PSN adds small random perturbations to a subset of these settings before generating an answer, keeping the perturbation constant for the same reasoning trajectory.
This is akin to having the same solver temporarily adopt a different set of thinking habits and then completing the problem using those habits. Parameter perturbations change selection tendencies at the source of the entire generation process, offering a better chance of forming coherent alternative reasoning paths compared to adding randomness word-by-word. However, it does not guarantee that every attempt will yield a new solution.
During training, the noisy model handles exploration, and the collected responses are used to update the unperturbed model. GRPO facilitates this by comparing a group of responses to the same problem, ensuring that better-performing responses receive more favorable learning signals.
Experiments compared different locations for noise injection. Among them, MLP/FFN modules—the feed-forward networks within each layer responsible for processing information—showed a good balance between exploration and stability.
For this method to work effectively, two supporting designs are required.
First, integrating exploration results back into normal training. Sampling comes from the noisy model, but updates are applied to the unperturbed model. The probabilities of generating the same response may differ between the two. Truncated Importance Sampling (TIS) adjusts sample weights based on this discrepancy and caps the weights to prevent a few outliers from skewing the training.
Second, adjusting noise based on the current state. A real-time scheduler first uses the unperturbed model to attempt each problem twice, observing the similarity of the responses and the concentration of the model's predictions. These two signals are combined; if they exceed the historical average, exploration is enhanced; otherwise, perturbations are reduced. This mitigates issues where fixed noise levels might be too strong or too weak for different problems.
PSN can also be combined with existing methods. When integrated with pass@k training, the average pass@256 improved from 76.37% to 79.12%, indicating that modifying parameters and modifying training objectives can complement each other.
Does It Work Across Different Models?
On Qwen2.5-Math-7B, the paper evaluated five mathematical benchmarks: AIME 2024, AIME 2025, AMC 2023, OlympiadBench, and Minerva. According to Table 3 in the main text, the average results for the adaptive version are as follows:
The most significant change appeared in multiple attempts: pass@256 increased by 4.8 percentage points, while single-attempt performance remained largely unchanged.
The authors also tested Qwen2.5-7B, Qwen3-4B-Base, and Llama-3.1-8B. With the fixed-noise version, the average pass@256 across the five math benchmarks for Qwen3-4B-Base rose from 70.70% to 74.56%, and for Llama-3.1-8B, it jumped from 39.56% to 49.92%. The adaptive version for Qwen3 further reached 76.10%. Detailed results for Qwen3-4B-Base are shown below, where "Clean" represents the standard GRPO algorithm, and "PSN" represents the method incorporating noise training (this paper).
Better Exploration Leads to Better Generalization
This corresponds to Out-of-Distribution (OOD) generalization: If training primarily uses mathematical data, can the capabilities transfer when tested on tasks from different domains? This is distinct from "testing on a different model."
The paper used the scientific QA benchmark GPQA Diamond and the programming benchmark LiveCodeBench to verify this. Taking Qwen3-4B-Base as an example, Appendix Table 7 reports that models trained with the PSN method demonstrate strong OOD generalization capabilities:
The PSN version of Llama-3.1-8B also showed improvements on these two tests. In other words, the benefits extended beyond the training domain, providing evidence that "this exploration mechanism is transferable."
Some prior methods, such as Pass@k[1] training proposed by ByteDance Seed and Renmin University of China, which modifies the optimization objective, also address the limited exploration capability of GRPO. Combining this paper's method with Pass@k training yielded even better results.
The table below shows that pass@k refers to the Pass@k training method proposed by ByteDance Seed and Renmin University of China.
Even the Entry Points for Problem-Solving Have Shifted
Improved scores alone do not directly answer a more intriguing question: Did the model devise entirely new methods, or did it simply select existing ones more frequently?
The authors sampled 300 responses for each of the 30 problems in AIME 2024. They filtered for questions where the baseline model failed all 300 times, but PSN succeeded at least once. They then used Gemini to perform a qualitative analysis of the reasoning processes.
The appendix provides an intuitive example: The vertices of a regular octagon are colored red and blue. The task is to determine if a rotation exists that maps all blue points onto the original positions of the red points.
Baseline models often fall into the trap of assuming "the pattern must be rotationally symmetric," prematurely excluding asymmetric coloring schemes. The image below illustrates common error patterns from various models after GRPO training:
In contrast, PSN approaches the problem from a different angle: The set of blue points after rotation has no overlap with the original set of blue points. This perspective unlocks solutions that were previously overlooked.
This observation aligns with the paper's concept of "new thinking modes": Solution angles absent from the baseline's 300 responses appear in PSN's outputs. While this supports the idea that the model explores previously unobserved paths, limited sampling cannot prove that the original model would never generate such insights.
The paper also examined the self-consistency of the reasoning process. Using ReCEval to measure whether earlier steps support subsequent conclusions and checking for internal contradictions, PSN-GRPO outperformed the compared baselines. As responses become more diverse, verifying coherence becomes equally critical.
Paper Title: Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026) Affiliations: Fudan University, Shanghai Institute for Advanced Intelligence, Shanghai AI Laboratory, The Chinese University of Hong Kong. arXiv: https://arxiv.org/pdf/2602.02555
Like, Share, and Bookmark
We welcome your thoughts in the comments!
— End —
[Academic Submissions] Please send emails on weekdays to: [email protected]. Include [Submission] in the subject line and tell us: Who you are, Where you come from, Your submission content, along with Project/Homepage links and Contact information.
🎓 We will try our best to reply promptly :)
🌟 Star Us 🌟
Daily Updates on Cutting-Edge Tech Progress