OpenAI 官方博客T1·Models
68 pts
OpenAI discovered that the 'gradient noise scale,' a simple statistical metric, effectively predicts the parallelizability of neural network training. The findings suggest that complex tasks with noisier gradients can benefit from larger batch sizes, removing potential limits on scaling AI systems. This work rigorizes model training, moving it away from being a mysterious art toward a systematic science.