Termsslug: inference-latency

Inference Latency

推理延迟

Time required from input request to generating the first or all tokens

Curated Intelligence

13 items · Page 1/2
InfoQ 中文T2·Models
58 pts
OpenAI Adds Ultrafast Tier to GPT-6.1 Sol: Token Price Up 6x, Performance Near Astra

OpenAI introduced an 'Ultrafast' service tier for the existing GPT-6.1 Sol model in its Responses API, enabling lower latency without changing the model name. While this mode significantly reduces delay, it costs 6 times more per token than the standard tier. OpenAI claims performance approaches GPT-6 Astra in complex coding and professional tasks, targeting users with high willingness to pay for speed.

Hugging Face BlogT1·Tools
68 pts
Hugging Face and Cerebras Launch Real-Time Voice AI with Gemma 4

Hugging Face and Cerebras have partnered to integrate the Gemma 4 model into a low-latency inference platform, creating an open, modular architecture for real-time voice interaction. The solution employs a cascaded speech processing approach to significantly reduce response times, addressing the critical latency bottleneck in current voice AI user experiences.

OpenAI 官方博客T1·Tools
68 pts
OpenAI Speeds Up Agent Workflows with WebSockets in Responses API

OpenAI introduced WebSocket support in the Responses API, reducing end-to-end latency for agentic workflows like Codex by 40%. As model inference speeds approach 1,000 tokens/s, API service overhead became the bottleneck; WebSockets mitigate this by minimizing HTTP handshakes and round-trips during tool execution loops. This update significantly enhances the developer experience for building real-time, high-frequency interactive agents.

OpenAI 官方博客T1·Infra
48 pts
OpenAI Partners with Cerebras for 750MW Compute

OpenAI has partnered with chipmaker Cerebras to add 750MW of high-speed AI compute, aiming to reduce inference latency and accelerate ChatGPT's real-time workloads. This move represents a strategic infrastructure collaboration focused on performance optimization.