Termsslug: tensor-parallelism

Tensor Parallelism

张量并行

Parallelism strategy that shards model weights and computation across GPUs by tensor dimension.

Curated Intelligence

11 items · Page 1/2
Hacker News AIT2·Infra
72 pts
Two DGX Stations Serve GLM-5.3 at 607.8 Tokens per Second

NVIDIA published a technical tutorial showing how to serve the 753B-parameter GLM-5.3 NVFP4 quantized MoE model locally across two DGX Stations linked via ConnectX-8 RDMA, using vLLM tensor parallelism (TP=2) to reach 607.8 tok/s through a single endpoint. The core challenge is that the 464 GB checkpoint exceeds the 252 GB HBM3e of a single GB300 GPU, so weights and computation are split across both GPUs with CPU offload disabled. The walkthrough offers a reproducible engineering path and concurrency benchmarking method for desk-side deployment of frontier-scale MoE models.

InfoQ 中文T2·Open Source
78 pts
StartLux Open-Sources Prism Inference Engine with Up to 5.3x Speedup

StartLux has open-sourced Prism, an inference engine optimized for decision models, achieving up to 5.3x speedup over vLLM on a single H200 GPU. Supporting hybrid TP/DP parallelism and deterministic outputs while maintaining BF16-level accuracy, Prism marks a shift in the domestic LLM stack towards specialized inference systems.

vLLM 官方博客T1·Models
78 pts
vLLM Day-0 Support for Reflection Beam 501B MoE Boosts Inference Speed

vLLM provides day-0 support for Reflection's Beam, a 501B-parameter MoE model. By integrating DFlash2 speculative decoding with adaptive verification, it achieves 708 tokens/s per request and boosts high-concurrency throughput by 10%. Additionally, agent-driven kernel optimization improved batch-1 decode speed by 2.71x and prefill by 1.87x.

vLLM 官方博客T1·Models
68 pts
vLLM Adds Day 0 Support for Meta's 816B MoE Muse Spark Open Model

vLLM and Meta Superintelligence Labs provide Day 0 inference support for Muse Spark Open, an open-weight sparse MoE model with 816B total parameters and a 1M context window designed for long-horizon agentic coding. The release features native multimodal perception and optimized parallelism strategies across Nvidia, AMD, and TPU hardware, including speculative decoding via a trained DSpark draft model.

vLLM 官方博客T1·Tools
68 pts
vLLM Optimizes Kimi K3 Serving: Achieving 2.8x Throughput via KDA Checkpoints and MXFP4 Kernels

The vLLM team achieved up to 2.8x throughput improvement and 72-85% lower TTFT for Kimi K3 on B300 nodes through full-stack optimizations. Key breakthroughs include adaptive speculative decoding budgets, internal KDA prefix checkpoints, zero-copy mixed batches, and deferred MXFP4 finalization. These changes address scheduling bottlenecks and small tensor copy overheads in the LatentMoE architecture, significantly enhancing inference efficiency under high concurrency.

vLLM 官方博客T1·Tools
78 pts
vLLM-Omni Integrates FastH3: Achieving Real-Time MiniMax H3 Video Generation via Full-Stack Optimization and Four-Step Diffusion

The vLLM-Omni framework implements system-wide full-stack optimizations for the MiniMax H3 model, covering attention mechanisms, communication, and parallel VAE decoding. It further integrates FastVideo's FastH3 technique to reduce DiT forward passes from 49 to just 4. Benchmarks on eight B300 GPUs demonstrate that this approach generates a complete 10.125-second MP4 video in approximately 8.7 seconds, achieving 'real-time' performance where generation speed exceeds playback duration.

vLLM 官方博客T1·Tools
78 pts
vLLM Introduces Decode Context Parallelism to Break MLA and GQA Memory Bottlenecks for Million-Token Agent Workloads

vLLM introduces Decode Context Parallelism (DCP) to address KV cache memory duplication bottlenecks in GQA and MLA architectures during long-context inference. By optimizing context distribution at the decode stage, DCP significantly enhances concurrency and memory efficiency for agent workloads ranging from 64K to 1M tokens on NVIDIA B200 GPUs.