Study on LLM RL Batch Scaling: Mechanisms for Accelerated Convergence and Failure Boundaries at High Throughput
A new study analyzes the non-linear impact of batch size on training efficiency in Large Language Model Reinforcement Learning (RL). Experiments show that after learning rate retuning, moderate batches (e.g., B, 2B, 4B) significantly reduce time-to-target by improving throughput. However, scaling up to 16B causes additional sample costs to outweigh throughput gains, resulting in slower convergence. This finding provides quantitative guidance for optimizing compute allocation in RLHF or RLAIF pipelines.
Related Entities
Full Report
!12 Figure 1. Think of training as climbing toward the same capability target. The summit is the shared validation target, each route is a different batch size, and each marker is one optimizer update. After learning-rate retuning, B, 2B, and 4B make similar progress per response, while higher throughput lets the larger batches reach the summit sooner. At 16B, however, the additional sample cost outweighs the throughput gain, so the apparently larger step becomes a slower route. The legend reports normalized time-to-target.! # Introduction Pretraining gives large language models a new capability baseline; reinforcement learning determines how far they can continue to advance by learning through exploration and feedback in new environments and tasks. DeepSeek-R1 helped demonstrate how large-scale RL can develop strong reasoning capabilities with little labeled data.[^r1] With Grok 4, xAI pushed the computational scale further: it reported using Colossus, its 200,000-GPU cluster, to run reinforcement learning at pretraining scale, with more than an order of magnitude more compute than in its previous RL work.[^grok] As RL becomes a larger part of model development, its efficiency has increasingly large consequences for training budgets. Consider an illustrative budget: 10,000 GPUs running continuously for 30 days, at an assumed USD 3 per GPU-hour, would cost USD 21.6 million. Reaching the same model quality in 10% less time would save 720,000 GPU-hours, or USD 2.16 million under those assumptions. For a team operating its own cluster, those GPU-hours become capacity for the next experiment. Where should we look for that gain? One place is a parameter that every training recipe already has: batch size. In practice, batch size enters two recurring decisions: - Batch-size tuning: adapting the recipe. A larger model or a different accelerator type can make the old batch size and learning rate unsuitable. A small batch may underutilize the hardware, while a mismatched learning rate may slow or destabilize learning. The goal is to find a batch size and matching hyperparameters that learn reliably while using the available hardware efficiently. - Batch-size scaling: turning parallelism into faster learning. If the same model receives more GPUs, can the batch grow with them and reach the target sooner? Even without more GPUs, can a larger generation batch use otherwise idle capacity? Both decisions lead to the same practical question: how do we find the batch size that reaches the target performance in the shortest wall-clock time? A larger batch can be either an accelerator or a speed bump. In our experiments, a moderately larger batch with learning-rate retuning preserves learning efficiency while improving throughput. But once the batch becomes too large, the additional sample cost can outweigh the throughput gain. Figure 1 depicts this contrast as different routes to the same summit: a larger batch often helps, but it does not always get the model to the target sooner. What scientific principle governs this transition from acceleration to slowdown, and can it be turned into a measurable criterion for selecting the batch size that minimizes time-to-target? To answer this question, we analyze batch scaling under the joint constraints of training dynamics and hardware execution, revisiting LLM RL scaling from first principles. This calls for the co-design of training algorithms and infrastructure. Ourpaperdevelops the resulting framework. # From Classical Batch Scaling to LLM Reinforcement Learning > How far can we usefully increase the batch? The modern large-batch story first became visible as an engineering breakthrough. In 2017, Goyal et al. trained ResNet-50 on ImageNet in one hour using 256 GPUs and a batch of 8,192, while preserving small-batch accuracy through linear learning-rate scaling and gradual warmup.[^goyal] The result showed that, with the right recipe, a larger batch could turn hardware parallelism into a dramatic reduction in wall-clock time. It also sharpened the next question: > how far can this scaling remain effective? McCandlish et al. (2018) connected the point of diminishing returns to the gradient noise scale.[^mccandlish] Shallue et al. (2019) showed that useful data parallelism depends on the workload and that fair batch-size comparisons require hyperparameter tuning.[^shallue] The practical implication is simple: doubling the samples per update does not necessarily halve the updates needed to reach the same target. A closely related question is whether changing the batch can preserve learning from a fixed number of samples. Hilton, Cobbe, and Schulman (2022) studied this as batch-size invariance for policy optimization: after adjusting the relevant hyperparameters, different batch sizes should produce similar learning behavior at equal sample counts.[^invariance] This is both a useful scaling intuition and a fair comparison principle: more samples should generally buy more capability, while the same cumulative sample budget should yield approximately the same capability across well-tuned batch sizes. > What changes in LLM RL? In supervised learning, examples already exist in a fixed dataset. Increasing the batch mainly changes gradient estimation, optimizer-update frequency, and training parallelism; the model does not have to produce its own training examples. LLM RL adds an online loop. A Transformer agent first uses the current policy to sample responses or interact with an environment, producing trajectories and rewards. Those newly generated data are then consumed by forward passes, backward passes, and optimizer updates; the updated policy generates the next batch. The model is therefore both the inference system that produces its training data and the learning system that consumes them.
!12 Figure 2. LLM reinforcement learning couples two GPU-intensive processes. Sampling uses autoregressive inference to produce responses and trajectories; training consumes those data to update the model. Batch size shapes both sides: generation batching determines how much inference work can run concurrently, while the training batch determines how much data each optimizer update consumes.! Both processes are GPU-intensive, but they exercise the hardware differently. Sampling generates one token at a time, repeatedly reads model weights, maintains KV caches, and schedules concurrent sequences. Training processes many token positions together but must store activations, compute gradients, and update optimizer states. As we show later, this computational asymmetry causes batch size to affect the two stages differently. # Two Efficiencies, One Framework Reaching the desired model quality sooner depends on two kinds of efficiency: - Sample efficiency measures how much the model learns per training response. Batch size and its associated hyperparameters determine the progress obtained from a given sample budget. - System efficiency measures how many training responses the full pipeline processes per second, accounting for generation, model updates, and waiting or synchronization. Let denote validation performance, cumulative training responses, and elapsed time. Along a smooth approximation to a learning trajectory, If throughput doubles but learning progress per response halves, the two effects cancel. Batch size can change both factors, so we evaluate them jointly. This decomposition provides the conceptual starting point for our fixed-hardware analysis. Actual learning curves are noisy and evaluated at discrete checkpoints, so we use time-to-target for the empirical comparison. Fix a validation target shared by all configurations. Let be the number of training responses needed to reach it and the average end-to-end throughput over that interval. Then Relative to a reference configuration, define the sample-cost ratio and throughput ratio . Here, means the new configuration needs more responses to reach the target (i.e., a sample penalty), while means it processes more responses per second (i.e., a throughput gain). ### The criterion: throughput must outweigh sample cost It is easy to see that a larger batch finishes sooner exactly when . ::: success Worked example Suppose the reference configuration needs 100,000 responses at 1,000 responses per minute, so it reaches the target in 100 minutes. A larger batch needs 120,000 responses but processes 1,500 responses per minute. Then , , and The larger batch is 20% faster even though it is less sample-efficient, because the 50% throughput gain exceeds the 20% sample penalty. If throughput improved by only 10%, the ratio would instead be , making the run about 9% slower. ::: This relationship gives two complementary views: - In theory, compare throughput gain with sample penalty. A larger batch reduces time-to-target only when . - In practice, establish invariance before optimizing throughput. Following the batch-size-invariance principle, retune batch-dependent hyperparameters and identify a range in which learning curves approximately align at equal cumulative response counts. Within that range, , so higher end-to-end throughput translates more directly into shorter time-to-target. ::: primary Takeaway Batch size affects both learning per response and end-to-end throughput. A larger batch reduces training time when its throughput gain outweighs any increase in the responses needed to reach the same quality. A simpler practical guideline is to first establish approximate batch-size invariance, then optimize throughput within that range. ::: # Step 1: Preserve Learning per Sample The first goal is to establish (approximate) batch-size invariance. Its implication is straightforward: doubling the batch should roughly halve the optimizer updates needed to reach the same model quality, leaving the total sample requirement approximately unchanged. After separately retuning batch-dependent hyperparameters, a fair and intuitive scaling comparison should satisfy two expectations: - More samples should generally improve performance. - At the same cumulative sample count, well-tuned runs with different batch sizes should achieve approximately the same performance. Unfortunately, this does not happen automatically. A larger batch averages gradients over more responses, but it also gives the model fewer opportunities to update within a fixed sample budget. Preserving learning per response therefore requires retuning how far each update moves the policy. Retune the learning rate. For Adam, we use the square-root rule as a starting point: Here, and denote the reference batch size and learning rate. The rule is motivated by prior adaptive-optimizer scaling analysis.[^adam] We test it empirically while holding other optimizer hyperparameters fixed; jointly tuning more batch-dependent hyperparameters may further extend invariance. To keep the comparison clean, we evaluate GRPO and PPO using our in-house training infrastructure. Both workloads use one global optimizer update per rollout batch, without minibatch subdivision or rollout reuse, avoiding additional tuning dimensions from minibatch size and optimization epochs. Because training is stochastic and evaluation is noisy, we expect batch-size invariance to appear as approximate alignment, rather than exact overlap, between empirical learning curves. ### Finding I. Approximate invariance holds over a substantial, bounded range GRPO: These experiments use Qwen3-30B-A3B-Instruct-2507 on a filtered subset of DAPO-MATH-17K, with responses per prompt. We evaluate mean@32 averaged over AIME 2024 and 2025. A training sample is a response retained after filtering and used in an optimizer update; each update contains retained prompt groups, giving . With learning-rate retuning, GRPO maintains approximately aligned sample-indexed learning curves across a 16-fold increase in prompt batch size. The left panel of Figure 3 shows this alignment from to over the measured training window. At and , the curves fall below this family at matched sample counts.
Figure 3. At fixed group size, retuned GRPO runs align over a bounded range of prompt batches. The largest batches depart from that family, and the decline in actor gradient norm flattens. The gray ±1-percentage-point band is a visual reference, not a confidence interval. The right panel of Figure 3 provides a complementary view of this departure. Across the approximately invariant range, actor gradient norms decrease roughly as . At and , this decline becomes shallower, coinciding with the loss of learning-curve alignment in the left panel. This pattern is consistent with diminishing benefits from gradient averaging: beyond this range, adding more responses to each update no longer compensates as effectively for taking fewer updates. PPO: We use an internal Hunyuan policy model and value model, each with 3B active parameters, trained on an internal dataset mixing math, science, and logic tasks. Evaluation performance is averaged across the corresponding benchmarks. Here, each prompt retains one response, so and .
Figure 4. Batch scaling in PPO. Left: validation performance versus cumulative training responses. Right: actor and critic gradient norms across batch sizes. The left panel of Figure 4 shows approximately aligned sample-indexed learning curves for batches from 256 through 2048, while batch 4096 falls below this family. This provides complementary evidence that learning-rate retuning can approximately preserve learning per response over a bounded batch range in PPO. The right panel reveals a difference between the actor and critic. Actor gradient norms decrease as the batch grows, with the decline becoming shallower at the largest batch. Critic gradient norms, by contrast, remain comparatively flat and noisy. These observations suggest that the actor and critic may have different useful batch scales. Given their distinct training objectives, understanding this difference and whether separate tuning can improve efficiency is an interesting direction for future work. ### Finding II. Total response count as the batch-size unit In GRPO, we can enlarge the batch along two axes: sampling more prompts or more responses per prompt. Because responses within a group share the same prompt and contribute to group-relative advantages, these two choices need not reduce variance in the same way. Do they nevertheless produce similar learning behavior when the total response batch is held fixed? We compare two pairs of configurations: versus , both with 1,024 responses per update, and versus , both with 2,048 responses per update. The figure denotes group size by , corresponding to in our notation. The left panel shows broadly similar learning trajectories, although the curves do not overlap at every checkpoint. The right panel shows a clearer pairing: configurations with the same total response batch have similar actor gradient norms, while the larger total batch produces consistently smaller norms. Together, these results support using total retained responses, , as the batch unit for the configurations tested.
Figure 5. Group-size comparison in GRPO. Left: validation performance versus cumulative training responses. Right: actor gradient norms over the same training window. Solid lines correspond to 1,024 responses per update; dashed lines correspond to 2,048. Blue and orange distinguish group sizes of 8 and 16. ### Finding III. Keeping the learning rate fixed breaks approximate batch-size invariance In practice, adding GPUs often increases both data parallelism and the global batch. If the learning rate and other hyperparameters remain tuned for the smaller batch, training may still look stable—and may even look smoother because gradient noise is lower. But at the same cumulative response count, the larger batch has taken fewer optimizer steps and may not reach the same quality. Stable training therefore does not imply batch-size invariance; the larger batch may need substantially more responses to catch up. We compare a reference configuration with and against two configurations with : one keeps the learning rate fixed, while the other increases it to following the square-root rule. Group size is held fixed.
Figure 6. Learning-rate control in GRPO. Left: validation performance versus cumulative training responses for the reference batch, a doubled batch with square-root learning-rate scaling, and a doubled batch with the learning rate fixed. Right: optimizer-update counts normalized to the reference first-hit count, illustrating how retuning brings the doubled batch closer to the ideal halving of updates. The left panel shows why retuning matters. With the scaled learning rate, the larger-batch run broadly follows the reference learning curve at matched sample counts. Keeping the learning rate fixed instead produces a substantial lag through much of training. Around 120K responses, for example, the fixed-learning-rate run scores approximately 74%, compared with roughly 77% for the reference and retuned runs. The right panel quantifies the difference in optimizer updates. Exact batch-size invariance predicts that doubling the batch should halve the updates needed to reach the same target. The scaled-learning-rate run has a target-crossing interval of 0.50–0.67× the reference first-hit count, consistent with the ideal 0.50× at the available checkpoint resolution. The fixed-learning-rate run instead requires 0.75–0.83×, substantially above that ideal. ::: primary Takeaway Learning-rate retuning approximately preserves learning per response over a bounded batch range in our GRPO and PPO experiments. In the GRPO group-size comparison, configurations with the same total response batch also show broadly similar learning trajectories and closely matched gradient norms. PPO likewise shows approximate invariance over a bounded range, but the actor and critic exhibit markedly different gradient-norm scaling as the batch grows. ::: # Step 2: Improve Throughput Even on Fixed Hardware Once learning per sample is approximately preserved, the next question is how to process those samples faster. A familiar approach is to add GPUs and increase data parallelism. But can a larger batch accelerate training without adding hardware? On the same devices, processing more responses per update might seem to simply take proportionally longer. In LLM RL, an opportunity comes from generation–training computational asymmetry. - During autoregressive decoding, each sequence produces one new token at a time. At low concurrency, the GPU repeatedly reads model weights to process relatively few token vectors. More active sequences allow more tokens to share those weight transfers, so the number of responses can grow faster than the time needed to generate them. - Training has a different execution pattern. Once responses are available, forward and backward passes process many token positions together, already exposing substantial parallelism. In this regime, increasing the batch mainly adds work. A simple local model summarizes this contrast: Here, counts responses in the logical rollout batch, while counts retained responses consumed by one optimizer update. Our experiments use asynchronous partial rollouts with , so increasing the training batch also increases the generation batch. The intercept captures generation cost that can be amortized across responses: increasing the batch reduces this cost per response, whereas training time grows approximately in proportion to its batch size. Ourpaperdevelops this intuition through a roofline view, explaining how batching changes arithmetic intensity and when execution is limited by memory bandwidth or compute. The simplified model above captures the resulting difference between generation and training costs. The throughput gain depends on how the execution system translates a larger generation batch into active decoding concurrency, subject to scheduling and memory constraints. We therefore measure end-to-end throughput using retained training responses per second, including time spent on discarded responses, model updates, and pipeline overhead. ### Finding IV. Response-collection time grows sublinearly over the measured range Figure 7 compares the increase in retained training responses with the increase in collection time, both normalized to the smallest batch. In both workloads, the blue response-count bars grow faster than the orange time bars. Their ratio gives the generation-stage throughput gain.
Figure 7. More responses need not take proportionally more collection time. On fixed hardware, a 4× batch increase raises generation-stage throughput by 2.29× in PPO and 1.36× in GRPO. In PPO, increasing the training batch from 256 to 1024 collects four times as many responses per optimizer update, while collection time increases from 39 to 68 seconds. Response-collection throughput therefore improves by . In GRPO, increasing from 128 to 256 improves generation-stage throughput by 1.31×; increasing it to 512 yields 1.36×. The small additional gain suggests that the available benefit is already flattening over this range. ::: primary Takeaway Generation and training scale differently with batch size: generation can amortize model-weight reads across more concurrent sequences, whereas training time grows approximately in proportion to the batch. When decoding has not yet saturated the hardware, doubling the generation batch generally increases collection time by less than 2×, improving generation throughput even without additional GPUs. ::: # Step 3: Combine Two Critical Batches to Minimize Time-to-Target The preceding results reveal two distinct limits to batch scaling in LLM reinforcement learning: - The critical generation batch concerns system efficiency. It marks where generation throughput approaches its plateau for a fixed model, decoding engine, response-length distribution, and hardware allocation. Below this scale, increasing the batch can improve hardware utilization; beyond it, further batching yields little additional generation throughput. - The critical training batch concerns sample efficiency. It marks the upper end of approximate batch-size invariance under learning-rate retuning. Within this range, larger batches need roughly the same number of responses to reach a given target. Beyond it, the benefit of averaging over more responses no longer fully compensates for taking fewer updates, increasing the sample requirement. These two limits need not coincide. Figure 8 illustrates their distinct roles, with memory capacity imposing a separate constraint on feasible generation batches. Their interaction determines how far increasing the batch can reduce training time.
Figure 8. Two critical batch sizes. Left: generation throughput approaches a plateau. Right: learning per response remains approximately preserved under learning-rate retuning before declining at larger batches. Both curves are schematic. ### From critical batches to training time How do these limits combine to determine training time? Consider a coupled configuration in which generation and training batches increase together. Figure 9 illustrates the case where generation throughput plateaus before learning efficiency deteriorates. In this schematic, the generation plateau also limits further gains in end-to-end throughput, producing a near-optimal range of training batch sizes.
!12 Figure 9. Schematic time-to-target when generation throughput saturates before sample efficiency deteriorates. The star marks the smallest batch on the near-optimal plateau.! This ordering produces three regimes: - Learning is preserved and throughput improves. Increasing the batch reduces training time because the model needs approximately the same number of responses, while the system processes them faster. - Learning is preserved but throughput plateaus. Further batching offers little additional speed. A range of batch sizes can therefore achieve similar training times. - The sample penalty dominates. Once larger batches require more responses to reach the target, limited throughput gains can no longer compensate, and training time increases. The ordering can also be reversed. If learning efficiency deteriorates before generation throughput plateaus, a larger batch can still finish sooner when its throughput gain outweighs the additional sample cost. ### Empirical Observations Figure 10 connects this reasoning to our GRPO experiments on fixed hardware, with and as the reference configuration.
Figure 10. Normalized GRPO training time across prompt batch sizes. Blue points use square-root learning-rate scaling; the orange control keeps the learning rate fixed. All times are normalized to the reference. The star marks the best measured configuration. With learning-rate retuning, increasing from 128 to 512 and 1024 reduces normalized time-to-target to 0.74× and 0.71×. At , the sample requirement remains unchanged while throughput improves by 41%, cutting training time from 11.90 to 8.42 hours—29% less time without adding GPUs. Larger batches do not always help. At and , a 60% increase in required responses outweighs throughput gains of 30% and 36%, raising training time to 1.23× and 1.18×. The deciding factor is whether throughput gains compensate for additional sample cost. # Conclusion The criterion translates into two practical stages: 1. Align learning per response. For each candidate training batch, retune the learning rate (and other hyper-parameters), compare learning curves at equal cumulative response counts, and estimate the sample-cost ratio at a shared validation target. 2. Measure the systems gain. Within memory limits, tune generation batching or decoding concurrency, measure end-to-end throughput , and select the feasible configuration that maximizes . Recheck whenever systems changes alter learning behavior.
# Future Work An interesting question is whether the useful batch range can be predicted, rather than rediscovered through expensive sweeps for every new setting. This requires a mechanistic understanding of why approximate batch-size invariance emerges, what determines its boundary, and why it eventually breaks. Learning-rate retuning is only a first step. A more complete scaling theory should explain how the relevant batch-dependent hyperparameters transfer across models, objectives, task distributions, and training stages. The PPO results further suggest that batch size may not have a single universal meaning even within one training algorithm. The actor and critic optimize different objectives and exhibit different gradient-norm scaling behavior; they may therefore have different critical training batches. Understanding these distinct dynamics could motivate separate batch sizes, update schedules, or tuning rules for the two components. At the systems level, our observations are specific to autoregressive Transformers, where token-by-token generation and parallel training place different demands on the hardware. Other architectures—including state-space, recurrent, or hybrid models—may shift this generation–training asymmetry and therefore have different critical batch scales. A model- and system-aware RL stack should account for these differences while decoupling generation concurrency from the optimizer batch, without overlooking the resulting trade-offs in queueing, policy staleness, and data freshness. The broader goal is therefore not merely to tune batch size, but to develop a predictive, architecture-aware understanding of RL scaling—one that jointly explains training dynamics, hardware execution, and their interaction in determining time-to-target. # Citation bibtex @misc{li2026largerbatches, title={When Do Larger Batches Help Scale LLM Reinforcement Learning?}, author={Ziniu Li and Jinbo Wang and Guanhua Huang and Feiyuan Zhang and Pengbo Li and Alex Chen}, year={2026}, eprint={2608.29296}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2608.29296} } [^goyal]: Goyal et al. (2017), Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. [^mccandlish]: McCandlish et al. (2018), An Empirical Model of Large-Batch Training. [^shallue]: Shallue et al. (2019), Measuring the Effects of Data Parallelism on Neural Network Training. JMLR. [^invariance]: Hilton, Cobbe, and Schulman (2022), Batch Size-Invariance for Policy Optimization. NeurIPS. [^adam]: Malladi, Lyu, Panigrahi, and Arora (2022), On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. NeurIPS. [^critical]: McCandlish et al. (2018), An Empirical Model of Large-Batch Training; Shallue et al. (2019), Measuring the Effects of Data Parallelism on Neural Network Training, JMLR. [^r1]: DeepSeek, DeepSeek-R1 Release, January 20, 2025. [^grok]: xAI, Grok 4: Scaling Up Reinforcement Learning, July 9, 2025.