NVIDIA published a technical tutorial showing how to serve the 753B-parameter GLM-5.3 NVFP4 quantized MoE model locally across two DGX Stations linked via ConnectX-8 RDMA, using vLLM tensor parallelism (TP=2) to reach 607.8 tok/s through a single endpoint. The core challenge is that the 464 GB checkpoint exceeds the 252 GB HBM3e of a single GB300 GPU, so weights and computation are split across both GPUs with CPU offload disabled. The walkthrough offers a reproducible engineering path and concurrency benchmarking method for desk-side deployment of frontier-scale MoE models.
Tensor Parallelism
张量并行
Parallelism strategy that shards model weights and computation across GPUs by tensor dimension.
Curated Intelligence
StartLux has open-sourced Prism, an inference engine optimized for decision models, achieving up to 5.3x speedup over vLLM on a single H200 GPU. Supporting hybrid TP/DP parallelism and deterministic outputs while maintaining BF16-level accuracy, Prism marks a shift in the domestic LLM stack towards specialized inference systems.
Hugging Face has launched a dedicated Space leveraging TorchNeuron to help developers verify if popular Transformers models run on AWS Trainium. The tool provides compatibility status, sharding details, and memory metrics for unmodified code, aiming to streamline the selection of cloud-native AI infrastructure.
vLLM provides day-0 support for Reflection's Beam, a 501B-parameter MoE model. By integrating DFlash2 speculative decoding with adaptive verification, it achieves 708 tokens/s per request and boosts high-concurrency throughput by 10%. Additionally, agent-driven kernel optimization improved batch-1 decode speed by 2.71x and prefill by 1.87x.
vLLM and Meta Superintelligence Labs provide Day 0 inference support for Muse Spark Open, an open-weight sparse MoE model with 816B total parameters and a 1M context window designed for long-horizon agentic coding. The release features native multimodal perception and optimized parallelism strategies across Nvidia, AMD, and TPU hardware, including speculative decoding via a trained DSpark draft model.
AWS introduces Ray LLM Deep Learning Containers enabling single LLM deployment across multiple GPU nodes on Amazon EKS. The solution exposes a unified OpenAI-compatible endpoint while natively supporting prefill/decode disaggregation to simplify distributed inference architecture setup and maintenance.
The vLLM team achieved up to 2.8x throughput improvement and 72-85% lower TTFT for Kimi K3 on B300 nodes through full-stack optimizations. Key breakthroughs include adaptive speculative decoding budgets, internal KDA prefix checkpoints, zero-copy mixed batches, and deferred MXFP4 finalization. These changes address scheduling bottlenecks and small tensor copy overheads in the LatentMoE architecture, significantly enhancing inference efficiency under high concurrency.
The vLLM team optimized MiniMax M3 on AMD Instinct MI355X, boosting output throughput by 3.14x to 342.4 tokens/s/GPU at concurrency 32 while reducing time-to-first-token. The key insight is a systematic 'follow the bottleneck' methodology for identifying dynamic optimization targets after initial model deployment.
The vLLM-Omni framework implements system-wide full-stack optimizations for the MiniMax H3 model, covering attention mechanisms, communication, and parallel VAE decoding. It further integrates FastVideo's FastH3 technique to reduce DiT forward passes from 49 to just 4. Benchmarks on eight B300 GPUs demonstrate that this approach generates a complete 10.125-second MP4 video in approximately 8.7 seconds, achieving 'real-time' performance where generation speed exceeds playback duration.
vLLM introduces Decode Context Parallelism (DCP) to address KV cache memory duplication bottlenecks in GQA and MLA architectures during long-context inference. By optimizing context distribution at the decode stage, DCP significantly enhances concurrency and memory efficiency for agent workloads ranging from 64K to 1M tokens on NVIDIA B200 GPUs.