OpenAI Chief Scientist Jakub Pachocki argues that progress in modern AI is fundamentally about training better compressors, i.e. predictors, and proposes perplexity (equivalently cross-entropy loss) as a comparable metric across model developers to underpin shared safety standards. He also pledges more transparency into OpenAI's pretraining progress and notes that recent models already show meaningful design contributions from AI.
Termsslug: cross-entropy-loss
Cross-Entropy Loss
交叉熵损失
The standard pretraining objective for language models, mathematically equivalent to perplexity and measuring divergence from th
Curated Intelligence
Hacker News AIT2·Models
78 pts
OpenAI 官方博客T1·Models
78 pts
OpenAI Chief Scientist Jakub Pachocki argues that progress in modern AI is fundamentally progress in training better compressors, i.e. predictors, and proposes perplexity (cross-entropy loss) as a shared, cross-developer metric to inform common safety standards. As a first step, OpenAI discloses more transparency into its pretraining progress, and notes that recent models already show meaningful design contributions from AI itself.