Today for AI
HOT RADAR
computeHEAT 9.3°

AI CURATED HOT STORY · 10/16/2026

HKUST and Huawei Propose BalanceRoute to Boost LLM Inference Throughput via Load Balancing

1 reports archived1 independent sourcesupdated 10/16/2026, 11:23:00
Synthesis & Latest Updates
1 Sources Cross-Validated

To address throughput bottlenecks caused by synchronization waits in data-parallel MoE deployments, HKUST and Huawei proposed BalanceRoute. By modeling load imbalance and converting short-term predictions into online routing decisions, the method improved decoding throughput by ~15% in a 96-GPU experiment. The work was accepted at NeurIPS 2026 and deployed in the MathBot-Pie system.

LATEST/To address throughput bottlenecks caused by synchronization waits in data-parallel MoE deployments, HKUST and Huawei proposed BalanceRoute. By modeling load imbalance and converting short-term predictions into online routing decisions, the method improved decoding throughput by ~15% in a 96-GPU experiment. The work was accepted at NeurIPS 2026 and deployed in the MathBot-Pie system.

HEAT TRENDHourly heat curve

3 fully observed hours
Now
9.3
Peak
9.820:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

TIMELINECoverage timeline

Total 1 reports · Latest first
  1. 机器之心 (微信公众号)T2·68 pts
    • In data-parallel MoE deployment, delays in the busiest compute group block others, causing diminishing throughput returns as scale increases.
    • BalanceRoute models synchronization wait metrics and converts request allocation into an online routing optimization problem based on short-term load prediction.
    • The method achieved ~15% higher decoding throughput compared to the strongest baseline in a 96-GPU test environment and is now deployed in production systems.
HKUST and Huawei Propose BalanceRoute to Boost LLM Inference Throughput via Load Balancing | Today for AI