NVIDIA Releases Three Methods to Speed Up MoE Inference on DGX Spark

NVIDIA introduced three optimization strategies for Mixture-of-Experts (MoE) inference on a single DGX Spark, validated using Ant Group's Ling-3.0-flash model. By reducing data movement per step and implementing speculative decoding, the approach achieved a 42.5% increase in decode throughput and a 24.5% reduction in mean latency on the GSM8K benchmark with negligible accuracy loss. This highlights methods to overcome memory bandwidth bottlenecks in small-batch scenarios, though specific module dependencies affect overall performance.