Together AI Launches Together Compiler for Agent-Driven Hot-Reloaded Inference
Together AI released Together Compiler, a system in which AI agents continuously re-tune the inference forward pass as production traffic shifts. Its core is a persistent serving engine that hot-reloads agent-written forward passes without restarts, backed by a 12-layer fast screening tier and a 60-layer slow tier that confirms gains on real traffic. Together claims up to 2.6x TPS over the best open configurations on SemiAnalysis InferenceX, though some gains are marginal and all figures are self-reported; the system is closed source and already live on its serverless endpoints.
Full Report
▲ Marktechpost AI · 2026-11-05 08
▲ Together AI just released Together Compiler, a system where AI agents keep re-tuning the inference forward pass as production traffic changes.
The key idea is a persistent engine. The serving engine starts once, and each agent-written forward pass is hot-reloaded with no restart. A 12-layer fast tier screens candidates, and a 60-layer slow tier confirms gains on real traffic. That gives agents 252x more experiments than restarting the engine each time.
On SemiAnalysis InferenceX, it beat the best open configurations by up to 2.6x TPS in under 24 hours per model. At batch size 1, MiniMax M3.1 Flash on B200 hit 1,111 P90 TPS per user vs 633. It is a trade-off though: some gains are small. GLM 5.3 on MI355X moved only from 463 to 480 TPS, and every number is self-reported.
It already runs on Together's serverless endpoints, with a limited beta for dedicated endpoints. Not open source. Tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash across B200, B300, GB300, MI355X and Trainium2......
Full analysis:
