Termsslug: gradient-noise-scale

Gradient Noise Scale

梯度噪声尺度

A statistical measure quantifying the difference between mini-batch gradient estimates and full-data gradients.

Curated Intelligence

1 items · Page 1/1
OpenAI 官方博客T1·Models
68 pts
OpenAI Proposes Gradient Noise Scale to Predict Training Parallelism

OpenAI discovered that the 'gradient noise scale,' a simple statistical metric, effectively predicts the parallelizability of neural network training. The findings suggest that complex tasks with noisier gradients can benefit from larger batch sizes, removing potential limits on scaling AI systems. This work rigorizes model training, moving it away from being a mysterious art toward a systematic science.