Best-of-N Alignment

The paper characterizes the information-theoretic limits of KL-regularized alignment, deriving a closed-form expression for the maximum achievable expected reward gain, which is governed by a Jeffreys divergence term rather than the √KL term used in prior analyses. The authors reformulate this as a covariance under the base model, yielding a practical estimator that predicts alignment gains from base-model samples alone, and extend the analysis to proxy rewards, showing reward hacking grows with reward error.