Two DGX Stations Serve GLM-5.3 at 607.8 Tokens per Second

1 reports archived1 independent sourcesupdated 11/14/2026, 22:09:25

Synthesis & Latest Updates

1 Sources Cross-Validated

NVIDIA published a technical tutorial showing how to serve the 753B-parameter GLM-5.3 NVFP4 quantized MoE model locally across two DGX Stations linked via ConnectX-8 RDMA, using vLLM tensor parallelism (TP=2) to reach 607.8 tok/s through a single endpoint. The core challenge is that the 464 GB checkpoint exceeds the 252 GB HBM3e of a single GB300 GPU, so weights and computation are split across both GPUs with CPU offload disabled. The walkthrough offers a reproducible engineering path and concurrency benchmarking method for desk-side deployment of frontier-scale MoE models.

LATEST/NVIDIA published a technical tutorial showing how to serve the 753B-parameter GLM-5.3 NVFP4 quantized MoE model locally across two DGX Stations linked via ConnectX-8 RDMA, using vLLM tensor parallelism (TP=2) to reach 607.8 tok/s through a single endpoint. The core challenge is that the 464 GB checkpoint exceeds the 252 GB HBM3e of a single GB300 GPU, so weights and computation are split across both GPUs with CPU offload disabled. The walkthrough offers a reproducible engineering path and concurrency benchmarking method for desk-side deployment of frontier-scale MoE models.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

Hacker News AIT2Nov 14, 22:09·72 pts
Two DGX Stations Serve GLM-5.3 at 607.8 Tokens per Second
- GLM-5.3 NVFP4 is a 753B-parameter MoE with 40B activated per token and a ~464 GB checkpoint, exceeding the 252 GB HBM3e of one GB300 GPU
- The setup links two DGX Stations over ConnectX-8 RDMA and splits weights and compute via vLLM TP=2 with CPU offload disabled
- Measured throughput reaches 607.8 tok/s on a single endpoint, with guidance on concurrency benchmarking and tuning
View Related Reports & Analysis Original Source