Anthropic has released Message Threads in public beta on the Claude platform, reducing time to first token (TTFT) by approximately 35% for long-context requests by transmitting only incremental changes instead of full conversation history. The feature benefits coding agents and document workflows across AWS, Azure, and Google Cloud, though total token usage remains unchanged.
Time to First Token (TTFT)
首字延迟
Latency from request initiation to first token generation, measuring responsiveness
Curated Intelligence
The CAICT released its 2026 MaaS Performance Monitoring Report, revealing that daily token consumption in China reached 500 trillion, nearly quadrupling since late 2025. While average tokens per second (TPS) doubled, time to first token (TTFT) increased due to larger model scales, with significant performance disparities observed between third-party and official platforms.
AWS publishes a technical guide demonstrating the deployment of the lightweight Kokoro-82M model on Amazon SageMaker AI. The solution addresses high latency in voice interactions by streaming text sentence-by-sentence to the TTS engine, significantly enhancing real-time responsiveness for AI agents.
Shi Technology unveiled TokenFleet, a distributed scheduling framework at the CCF HPC conference, designed to boost effective output of domestic compute clusters via optimized request routing, cross-datacenter capacity coordination, and mixed workload governance. Featuring three core technologies—TokenPilot, FleetMind, and SLOMax—the framework claims to reduce Time-to-First-Token (TTFT) latency by 32.4% while improving cost-efficiency per unit of valid output.
AI infrastructure firm Infinigence AI has completed a 2.3 billion RMB Series C round, bringing its total funding to 4.3 billion RMB and setting a new record for China's AI-native infrastructure sector. Specializing in the 'middle layer' via an M×N paradigm, the company enables heterogeneous chip mixed training and cross-domain scheduling; its proprietary PDD architecture reduces Time-To-First-Token latency by 51.5% and per-token costs by 37.5%. The company has also confidentially filed for an HKEX IPO, targeting a listing in the first half of 2027.
vLLM implements bi-directional KV cache transfer in its Prefill-Decode disaggregation architecture to reduce Time To First Token (TTFT) latency for multi-turn agentic workloads. This allows the decode node to return generated token states to the prefill node for reuse, eliminating redundant computation of accumulated context.
Alibaba's Qoder introduced a Fast mode for its desktop client, reducing first-response latency to 3 seconds while cutting credit consumption to 0.8x. The update optimizes inference efficiency, with benchmarks showing it completes common dev tasks roughly three times faster than domestic rivals and four times faster than overseas competitors.
The vLLM team has released vllm-metal, bringing vLLM's scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon. By leveraging MLX and Metal for execution, it addresses concurrency issues like Time to First Token (TTFT) and memory growth in local Mac inference. The current v0.29.0 version supports Homebrew installation and introduces batched multi-token prediction (MTP) along with GGUF model support.