NVIDIA Releases Three Methods to Speed Up MoE Inference on DGX Spark
NVIDIA introduced three optimization strategies for Mixture-of-Experts (MoE) inference on a single DGX Spark, validated using Ant Group's Ling-3.0-flash model. By reducing data movement per step and implementing speculative decoding, the approach achieved a 42.5% increase in decode throughput and a 24.5% reduction in mean latency on the GSM8K benchmark with negligible accuracy loss. This highlights methods to overcome memory bandwidth bottlenecks in small-batch scenarios, though specific module dependencies affect overall performance.
Full Report
▲ Marktechpost AI · 2026-11-03 20
▲ NVIDIA just published 3 ways to speed up MoE inference on a single DGX Spark, tested on Ant Group inclusionAI's 124B Ling-3.0-flash.
The key idea: at batch size 1, decoding is memory-bound. Reading weights, not math, sets the pace. So NVIDIA cut the bytes moved per step (Humming MoE kernels, an FP8 LM head) and the steps per answer (built-in MTP speculative decoding).
On all 1,319 GSM8K questions, decode throughput rose 42.5% (37.81 to 53.89 tok/s) and mean latency fell 24.5%, with accuracy flat (96.89% vs 96.82%). It is a trade-off though: the +185.1% synthetic gain does not carry over, as MTP acceptance drops from 3.90 to 2.68 tokens per round. And FP8 LM head + MTP need a shared LM-head module, or speed falls below baseline.
Ling-3.0-flash activates 5.1B of 124B parameters per token, ships under MIT, and the DGX Spark recipe is in the Ling cookbook......
Full analysis:
