NVL72 Clusters Need Topology-Aware Scheduling to Boost Usable GPUs by 27%

Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.

Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.

However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.

This blog covers the communication bottleneck in large-scale training and inference, the NVL72 rack-scale fabric, and slice sizing for resilience to GPU failures, as well as how the SkyPilot Platform keeps multi-node jobs rack-local through topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters.

Training at scale is a communication problem #

Frontier training jobs run across thousands of GPUs. Doing so requires answering two questions:

1. How do you shard your model across GPUs to ensure that it can actually run?
2. Given your compute budget, how many copies of the sharded model can you run to maximize training throughput?

The first question is typically answered with a combination of:

• Tensor Parallelism (TP): splitting a layer’s weight matrices across GPUs
• Expert Parallelism (EP): placing a mixture-of-experts (MoE) layer’s experts on different GPUs
• Pipeline Parallelism (PP): splitting a model layer-wise across GPUs

The answer to the second question sets the degree of Data Parallelism (DP): the number of copies of the sharded model that train in parallel, each on a different slice of the training data.

Of these parallelism strategies, tensor parallelism’s all-reduce and expert parallelism’s all-to-all communication patterns sit on the critical path during training, making them the most important to keep on NVLink (§9.1.2).

Tensor Parallelism

Tensor parallelism splits the weight matrices inside each layer across GPUs, so every GPU only stores and computes a slice of each layer.

Transformer layers are made up of two blocks: self-attention and a multi-layer perceptron (MLP). At its core, each block is two matrix multiplications (GEMMs) in a row. Megatron-LM (§3) showed that by partitioning a block’s first GEMM column-wise and its second GEMM row-wise, the GPUs only need to synchronize via all-reduce once per block, after the second GEMM, instead of after each one.

Figure 1. Tensor parallelism across two GPUs. A is split column-wise, so each GPU computes its own slice of y without communicating. B is split row-wise to match, so each GPU produces a partial z, and one all-reduce sums the partial results.

Even with this optimization, tensor parallelism still requires two all-reduces per transformer layer during the forward pass. These all-reduce steps sit on the critical path during training because they cannot be fully overlapped with computation. As a result, tensor parallelism is typically kept on NVLink: Hugging Face measured a 43% drop in training throughput when it moved off NVLink onto InfiniBand.

Expert Parallelism

In a mixture-of-experts (MoE) layer, the MLP block is replaced by many smaller MLPs, called experts, and a router that sends each token to a small subset of experts. Expert parallelism splits MoE layers such that these small, independent experts are placed on different GPUs.

Under expert parallelism, each MoE layer requires two all-to-all communication steps during the forward pass, dispatch and combine. A router selects the top-k experts for each token and dispatches tokens to the GPUs that have those experts in memory. Each GPU then computes its local experts on the tokens it received via a single grouped matrix multiplication. Finally, the expert outputs are sent back to each token's original GPU and combined.

Figure 2. Expert parallelism distributes experts across GPUs. The all-to-all communication dispatches tokens to their assigned experts and combines results. (Source: NVIDIA, Figure 4.)

As models add more MoE layers and route each token to more experts, the volume of all-to-all traffic grows, introducing a communication bottleneck.

Training an MoE model with 64-way expert parallelism spreads each MoE layer’s experts across 64 GPUs. On standard 8-GPU servers, that moves the all-to-all traffic off high-bandwidth NVLink and through much narrower InfiniBand interconnects where unoptimized communication can consume up to 60% of training time (§1.2). On NVL72 systems, all 64 GPUs stay within a single NVLink domain, effectively eliminating the communication bottleneck (§9.2.2).

Inference needs fast inter-node networking too #

The parallelism techniques discussed above are not unique to training. Beyond enabling inference engines to serve models that are too large for a single GPU, model parallelism can also improve throughput. Splitting a model's weights across more GPUs frees up memory for the KV cache, so the engine can process more concurrent requests or support larger context windows. In some cases, the freed memory grows faster than the number of GPUs: in one vLLM benchmark, splitting a model tensor-parallel across two GPUs increased the amount of KV cache blocks by 13.9x and delivered 3.9x more throughput.

Figure 3. GPU memory breakdown for Llama-3.1-70B on one vs. two H100s. (Data: vLLM.)

Expert parallelism follows the same logic. With a low degree of expert parallelism, each GPU holds many experts, leaving less room for the KV cache. WideEP, which refers to expert-parallel configurations spanning multiple nodes, significantly reduces the number of experts per GPU. The trade-off is network traffic: every all-to-all now spans more GPUs. On standard 8-GPU servers, an expert-parallel group can span at most eight GPUs before it has to communicate over InfiniBand, where per-GPU bandwidth drops 18x. NVL72 removes this limit by keeping all 72 GPUs in a rack on NVLink. When serving DeepSeek-R1 at interactive speeds, GB200 NVL72 with 32-way expert parallelism delivers up to 4.4x higher throughput per GPU than 8-GPU B200 servers, which are capped at 8-way.

The rack is the new server #

NVL72 extends NVLink from a single node to the entire rack, so the whole rack acts like a gigantic accelerator with shared memory.

Figure 4. The NVLink hierarchy within an NVL72 rack. (Source: NVIDIA, Figure 1.)

Every GPU on the NVL72 rack has 18 NVLink ports each delivering 100 GB/s of bidirectional bandwidth, which sums up to a total of 1.8 TB/s for each GPU. Each NVLink port on a given GPU maps to one of 18 NVSwitch chips that power the rack's fabric backplane. Because each NVSwitch chip accepts 72 NVLink lines, one from each of the GPUs, the aggregate internal bandwidth within the rack is 130 TB/s.

Figure 5. Each GPU connects one NVLink port to each of the rack's 18 NVSwitch chips. (Source: NADDOD.)

This network topology ensures that the fabric is fully non-blocking, meaning that any GPU in the rack can transmit data to any other GPU with uniform latency, making it ideal for large-scale distributed AI workloads.

Figure 6. The NVSwitch fabric is fully non-blocking. Any GPU can communicate with any other GPU in the rack at full bandwidth with the same latency. (Source: NVIDIA, Figure 1)

Sometimes less is more: using less nodes in each rack significantly improves reliability #

With NVL72 systems, it is tempting to allocate entire 18-node racks to workloads to maximize cluster utilization. In practice, teams actually get more usable capacity out of their cluster by leaving a couple of spare nodes in each rack.

GPUs are unreliable. On its gigawatt-scale supercluster, Meta measured a mean time between interruptions (MTBI) of roughly 490 days per 4-GPU node after putting significant engineering effort into improving reliability. NVL72 clusters make the 72-GPU rack the atomic unit of compute, which increases the fault blast radius by 18x compared to the 4-GPU nodes that make up the rack. The MTBI for the rack is given by:

Loading...

Based on the formula above, a rack would be interrupted roughly every 490 / 18 ≈ 27 days. If jobs are allocated entire 18-node racks, each of those interruptions makes the entire rack unschedulable until the failed node is repaired. On a 128-rack cluster, this means that a rack would become unusable for full-rack jobs roughly every 5 hours.

Reserving two nodes per rack as spares changes this. When a node fails, a spare in the same rack replaces it, so a single node failure no longer takes the entire rack out of commission. The cost is that two nodes per rack sit idle while everything is healthy.

To quantify this trade-off, NVIDIA modeled how different slice sizes affect usable capacity on a 128-rack (2,304-node) cluster as the share of unavailable nodes grows.

Figure 7. Usable capacity of a 128-rack cluster for different slice sizes as node unavailability increases. Each slice must fit within a single rack. (Adapted from NVIDIA, Figure 6.)

At SkyPilot, we manage tens of thousands of our customers’ GPUs, and on production NVL72 clusters we have measured unavailability rates of around 2%. At that rate, NVIDIA’s model shows that 16-node slices provide 27% more usable capacity than 18-node slices (2,037 vs. 1,602 nodes). They also beat 9-node slices by 4%, even though two 9-node slices fill a rack exactly: with 9-node slices, every failure costs half a rack, while with 16-node slices, the first two failures in a rack are fully absorbed. Compared with 8-node slices, 16-node slices give each job twice as many GPUs per NVLink domain while giving up less than 1% of usable capacity until node unavailability exceeds about 3.3%.

Sixteen nodes is also a natural size for distributed AI workloads: 64 GPUs is a power of two, which matches how parallelism groups are typically sized.

How a modern scheduler keeps jobs on NVLink #

So far, we have covered why the most communication-heavy parallelism techniques belong on NVLink, how NVL72 extends NVLink from a single node to the whole rack, and why 16-node slices with two spare nodes per rack get the most usable capacity out of production clusters. SkyPilot puts this into practice by placing each slice on a single rack, recovering from hardware failures in-place, and keeping whole racks free as jobs come and go.

Schedulers need to understand the hardware

Choosing the right slice size is only the first step. How do researchers ensure that each slice of a multi-node training run actually lands on a single rack?

Naive schedulers split multi-node jobs across racks, forcing workers to communicate over InfiniBand (400 Gb/s per direction) instead of NVLink (1.8 TB/s bidirectional):

Figure 8. Naive placement splits a multi-node job across racks.

SkyPilot’s Topology-Aware Scheduler (TAS) ensures that workers are colocated on the same rack:

Figure 9. SkyPilot's Topology-Aware Scheduler keeps the job's workers on a single rack.

For multi-rack workloads, TAS splits the job into 16-node (configurable) slices, and places them each on their own rack with two spare nodes. Then, when SkyPilot’s GPU Manager detects GPU failures, SkyPilot Frontier Trainer can hot swap the faulty node with one of the reserved healthy nodes while the remaining nodes continue running without interruption. If there are no remaining spares in the same rack, Frontier Trainer will reschedule the slice on the faulty rack to a healthy rack:

Figure 10. Slice-based scheduling holds two healthy spare nodes in each rack. When GPU Manager detects a failed node, Frontier Trainer swaps the faulty node for a spare in the same rack.

To quantify the impact of slice-based scheduling, NVIDIA’s GR00T team compared the effect of two different scheduling policies during pre-training of GR00T N1.7, a multimodal foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes. The first run used a naive scheduler that placed nodes across an arbitrary number of racks. The second run was scheduled with a rack-aware, slice-based policy like the one in Figure 10, which placed it on exactly 8 racks. Rack-aware placement raised throughput by 14.4%, cutting each iteration from 1.59 s to 1.39 s.

On a 10,000-GPU cluster, the same speedup saves roughly $16.7K per training hour. That’s about $400K saved for every training day, just by making better scheduling decisions.

Fragmentation blocks rack-scale jobs

So far we have explored how topology-aware scheduling works in isolation. In practice, large clusters are typically shared: heterogeneous workloads such as pre-training runs, RL rollouts, and batch inference all compete for the same resources. TAS bin packs workloads within the same rack at scheduling time, but jobs finish at different times, leaving free nodes scattered across racks. This is fragmentation: the cluster has enough free GPUs to run a rack-scale job, but because they’re spread across racks, the job can’t be scheduled.

Figure 11. Jobs finish at different times, leaving free nodes split across racks. With 16 nodes free but split between the two racks, the 16-node hero-run cannot be placed and waits.

SkyPilot’s Automatic Defragmentation (Auto Defrag) is built to address this. When a job is blocked by topology rather than capacity, Auto Defrag finds the least disruptive set of workloads to preempt to free up an entire rack. Those workloads are rescheduled on other racks, allowing the blocked job to be scheduled.

Figure 12. Auto Defrag preempts sft-d and reschedules it in rack-a, where it resumes from checkpoint. This frees up rack-b, so hero-run can be scheduled as a single 16-node slice.

Auto Defrag only preempts workloads that restart safely, such as managed jobs that resume from a checkpoint. Interactive development environments are left alone unless an admin explicitly opts them in.

Conclusion #

Large-scale AI workloads are distributed by default, and their most network-intensive communication patterns sit on the critical path. Rack-scale systems like NVL72 aim to address this bottleneck in hardware, but the hardware alone doesn't guarantee the speedup. These systems require rack-aware placement, resilience to hardware failures, and dynamic rebalancing as jobs come and go. The SkyPilot Platform is purpose-built to close that gap: Topology-Aware Scheduling keeps multi-node jobs on NVLink, Frontier Trainer swaps in a healthy spare when GPU Manager detects hardware failures, and Automatic Defragmentation keeps shared racks open for large jobs. If you're wrangling NVL72 clusters, we'd love to chat.