Together AI released Together Compiler, a system in which AI agents continuously re-tune the inference forward pass as production traffic shifts. Its core is a persistent serving engine that hot-reloads agent-written forward passes without restarts, backed by a 12-layer fast screening tier and a 60-layer slow tier that confirms gains on real traffic. Together claims up to 2.6x TPS over the best open configurations on SemiAnalysis InferenceX, though some gains are marginal and all figures are self-reported; the system is closed source and already live on its serverless endpoints.
Batch Size
批次大小
The number of samples used in one iteration of parameter update.
Curated Intelligence
A new study analyzes the non-linear impact of batch size on training efficiency in Large Language Model Reinforcement Learning (RL). Experiments show that after learning rate retuning, moderate batches (e.g., B, 2B, 4B) significantly reduce time-to-target by improving throughput. However, scaling up to 16B causes additional sample costs to outweigh throughput gains, resulting in slower convergence. This finding provides quantitative guidance for optimizing compute allocation in RLHF or RLAIF pipelines.
OpenAI discovered that the 'gradient noise scale,' a simple statistical metric, effectively predicts the parallelizability of neural network training. The findings suggest that complex tasks with noisier gradients can benefit from larger batch sizes, removing potential limits on scaling AI systems. This work rigorizes model training, moving it away from being a mysterious art toward a systematic science.