OpenAI 官方博客T1·Models
78 pts
OpenAI researchers identified a significant divergence between true human preferences and proxy reward model scores as policy optimization deepens in RLHF. The study quantifies scaling laws for this 'overoptimization' phenomenon, highlighting KL divergence constraints as a key mitigation strategy, providing critical theoretical grounding for reward modeling in AI alignment.