Termsslug: overoptimization

Overoptimization

过优化

Phenomenon where excessive fitting to proxy objectives degrades true performance

Curated Intelligence

1 items · Page 1/1
OpenAI 官方博客T1·Models
78 pts
OpenAI Reveals Scaling Laws for Reward Model Overoptimization

OpenAI researchers identified a significant divergence between true human preferences and proxy reward model scores as policy optimization deepens in RLHF. The study quantifies scaling laws for this 'overoptimization' phenomenon, highlighting KL divergence constraints as a key mitigation strategy, providing critical theoretical grounding for reward modeling in AI alignment.