NVIDIA achieved a 1.68x inference speedup for the 2.8T-parameter Kimi K3 model on GB300 NVL72 hardware via upstream vLLM optimizations, reaching 423 tok/s/user. The breakthrough relies on a massive 12,000-line Megakernel written in NVIDIA's new Cute DSL, demonstrating the future of AI-assisted generation of advanced compute kernels.
Operator Fusion
算子融合
Combining multiple consecutive operations into a single kernel to reduce memory access and launch overhead
Curated Intelligence
Moonshot AI leveraged agent-assisted development in vLLM to build a unified CuTe DSL decode path for Kimi K3. The optimization achieves over 840 tokens/s at batch 1 on NVIDIA GB300 clusters and boosts throughput by 1.5–2x in tested configurations, marking a significant step in using AI to write low-level GPU kernels.
Hugging Face releases the second part of its PyTorch profiling series, demonstrating how to use profiler traces to optimize standard nn.Linear layers into fused MLPs. The article provides a concrete case study on identifying computational bottlenecks and implementing operator fusion for improved inference efficiency.
The Qwen team has open-sourced FlashQLA, a high-performance linear attention library built on TileLang and optimized for Gated Delta Networks (GDN). By leveraging gate-driven intra-GPU sequence parallelism and hardware-friendly algebraic rewrites, it achieves 2-3x forward and 2x backward speedups on NVIDIA Hopper GPUs, significantly reducing training and inference costs for long-context models.