ALL INTELLIGENCE
OpenAI 官方博客T1·Models

OpenAI Publishes Foundational Research on Learning from Human Preferences

Original title: Learning from human preferences
65AI Score

OpenAI published the paper 'Learning from Human Preferences' in 2017, introducing a method to train reward models using human feedback. This research established the Reinforcement Learning from Human Feedback (RLHF) paradigm, providing the core theoretical foundation for subsequent alignment and instruction tuning of large language models.

Full Report

Contents4 sections

Skip to main content

Log inTry ChatGPT(opens in a new window)

Learning from human preferences | OpenAI

June 13, 2017

Release

Learning from human preferences

Read paper(opens in a new window)

Image 1: Learning From Human Preferences

Listen to article 4

Audio 2

Share

Challenges

One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.

We present a learning algorithm that uses small amounts of human feedback to solve modern RL environments. Machine learning systems with human feedbackhave⁠(opens in a new window)been⁠(opens in a new window)explored⁠(opens in a new window)before⁠(opens in a new window), but we’ve scaled up the approach to be able to work on much more complicated tasks. Our algorithm needed 900 bits of feedback from a human evaluator to learn to backflip—a seemingly simple task which is simple to judge butchallenging⁠to specify.

Image 2: Humanfeedbackjump

Our algorithm learned to backflip using around 900 individual bits of feedback from the human evaluator.

The overall training process is a 3-step feedback cycle between the human, the agent’s understanding of the goal, and the RL training.

Image 3: Diagram2x 2

Our AI agent starts by acting randomly in the environment. Periodically, two video clips of its behavior are given to a human, and the human decides which of the two clips is closest to fulfilling its goal—in this case, a backflip. The AI gradually builds a model of the goal of the task by finding the reward function that best explains the human’s judgments. It then uses RL to learn how to achieve that goal. As its behavior improves, it continues to ask for human feedback on trajectory pairs where it’s most uncertain about which is better, and further refines its understanding of the goal.

Our approach demonstrates promising sample efficiency—as stated previously, the backflip video required under 1000 bits of human feedback. It took less than an hour of a human evaluator’s time, while in the background the policy accumulated about 70 hours of overall experience (simulated at a much faster rate than real-time.) We will continue to work on reducing the amount of feedback a human needs to supply. You can see a sped-up version of the training process in the following video.

Image 4: Learning From Human Preferences Poster

00

We’ve tested our method on a number of tasks in the simulated robotics and Atari domains (without being given access to the reward function: so in Atari, without having access to the game score). Our agents can learn from human feedback to achieve strong and sometimes superhuman performance in many of the environments we tested. In the following animation you can see agents trained with our technique playing a variety of Atari games. The horizontal bar on the right hand side of each frame represent’s each agents prediction about how much a human evaluator would approve of their current behavior. These visualizations indicate that agents trained with human feedback learn to value oxygen in Seaquest (left), anticipate rewards in Breakout and Pong (center), or work out how to recover from crashes in Enduro(right).

Image 5: Seaquestsave

Image 6: Spaceinvadersbehavior

Image 7: Pong2

Image 8: Enduro1

Note there’s no need for the feedback to align with the environment’s normal reward function: we can, for example, train our agents to precisely keep even with other cars in Enduro rather than maximizing game score by passing them. We also sometimes find that learning from feedback does better than reinforcement learning with the normal reward function, because the human shapes the reward better than whoever wrote the environment’s reward.

Challenges

Our algorithm’s performance is only as good as the human evaluator’s intuition about what behaviors look correct, so if the human doesn’t have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it, as shown below.

Image 9: Gifhandlerresized

We addressed this particular problem by adding in visual cues (the thick white lines in the above animation) to make it easy for the human evaluators to estimate depth.

Footnote

Image 10: Goodgifb1 2

Image 11: Badgif3

By comparison, we took two hours to write our own reward function (the animation in the above right) to get a robot to backflip, and though it succeeds it’s a lot less elegant than the one trained simply through human feedback (top left). We think there are many cases where human feedback could let us specify a specific goal more intuitively and quickly than is possible by manually hand-crafting the objective.

You can replicate this backflip ingym⁠(opens in a new window)with the following reward function for Hopper:

Python

1def reward_fn(a, ob):2 backroll = -ob[7]3 height = ob[0]4 vel_act = a[0] * ob[8] + a[1] * ob[9] + a[2] * ob[10]5 backslide = -ob[5]6 return backroll * (1.0 + .3 * height + .1 * vel_act + .05 * backslide)

Authors

Dario Amodei, Paul Christiano, Alex Ray