ALL INTELLIGENCE
OpenAI 官方博客T1·Models

OpenAI Reveals Scaling Laws for Reward Model Overoptimization

Original title: Scaling laws for reward model overoptimization
78AI Score

OpenAI researchers identified a significant divergence between true human preferences and proxy reward model scores as policy optimization deepens in RLHF. The study quantifies scaling laws for this 'overoptimization' phenomenon, highlighting KL divergence constraints as a key mitigation strategy, providing critical theoretical grounding for reward modeling in AI alignment.

Full Report

Contents2 sections

Skip to main content

Log inTry ChatGPT(opens in a new window)

Try ChatGPT(opens in a new window)Login

OpenAI

October 19, 2022

Publication

Scaling laws for reward model overoptimization

Read paper(opens in a new window)

Image 1: Scaling Laws For Reward Model Overoptimization

Loading…

Audio 2

Share

Abstract

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-n sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

Authors

Leo Gao, John Schulman, Jacob Hilton