Tsinghua's TokenRouter Boosts LLM Serving Throughput by 64x via Per-Token Model Routing

1 reports archived0 independent sourcesupdated 10/10/2026, 11:32:00

Synthesis & Latest Updates

Tsinghua University introduced TokenRouter, a serving system that dynamically routes between small and large models at the token level to eliminate blocking delays in traditional frameworks. By enabling independent model deployment with shared KV caches and request buffering for optimized batching, it achieves throughput improvements ranging from 2.01x to 64.15x across five routing strategies.

LATEST/Tsinghua University introduced TokenRouter, a serving system that dynamically routes between small and large models at the token level to eliminate blocking delays in traditional frameworks. By enabling independent model deployment with shared KV caches and request buffering for optimized batching, it achieves throughput improvements ranging from 2.01x to 64.15x across five routing strategies.

HEAT TRENDHourly heat curve

54 fully observed hours

Now

0.0

Peak

16.113:00

24h change

-100%

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

TIMELINECoverage timeline

Total 1 reports · Latest first

Rohan Paul (@rohanpaul_ai)T2·Oct 10, 11:3278 pts
Tsinghua's TokenRouter Boosts LLM Serving Throughput by 64x via Per-Token Model Routing
- TokenRouter enables dynamic model routing at the token level, breaking the single-model-per-request limitation.
- Decoupling model servers while sharing KV caches eliminates synchronization overhead where fast models wait for slow ones.
- Experiments show up to 64.15x throughput improvement over existing state-of-the-art configurations across various routing strategies.
View Evidence Original Source