返回全部资讯
OpenAI 官方博客T1·基座大模型

OpenAI 提出梯度噪声尺度预测训练并行性

原标题:How AI training scales
68AI 研判分

OpenAI 发现“梯度噪声尺度”这一统计指标能有效预测神经网络训练的并行化潜力。研究表明,复杂任务通常伴随更高的梯度噪声,这意味着未来可通过增大批次规模来突破算力瓶颈,推动 AI 系统持续扩展。该成果将模型训练从依赖经验的“玄学”转化为可量化、系统化的科学过程。

报道正文

We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one potential limit to further growth of AI systems. More broadly, these results show that neural network training need not be considered a mysterious art, but can be rigorized and systematized.