DeepSeek unveiled DeepInfer at PyTorch Conf, an open-source inference engine claiming 155T tokens/day per cluster. It aims to challenge closed-source infrastructure dominance by offering a high-throughput, low-latency SOTA alternative.
Throughput
吞吐量
The amount of data or requests processed by a system per unit time
Curated Intelligence
MiniMax officially released M3.1 Flash, the revealed identity of the anonymous 'Space Bunny' model. It scored 29.6% on Agents’ Last Exam (ALE), ranking first among domestic models, and entered the global top tier in coding, security, and Computer Use tasks. With output speeds exceeding 200 TPS and API pricing at just 30% of its predecessor, it offers significant cost-performance advantages.
Rails creator DHH announced his team has ceased manual coding, using AI agents to rewrite the Campfire app from Ruby into Rust, Go, and Elixir. His benchmarks claim the Rust version achieves 150x higher throughput than Rails in chat room loading, advocating a paradigm where humans no longer read code. However, the results faced significant community backlash due to lack of implementation details and suspected measurement flaws.
This tool estimates GPU requirements for processing millions to trillions of tokens, citing ~367 H100s for 1 trillion monthly tokens on Llama 3.3 70B. It derives results from throughput, utilization, and I/O ratios, highlighting sensitivity to key assumptions.
A new study analyzes the non-linear impact of batch size on training efficiency in Large Language Model Reinforcement Learning (RL). Experiments show that after learning rate retuning, moderate batches (e.g., B, 2B, 4B) significantly reduce time-to-target by improving throughput. However, scaling up to 16B causes additional sample costs to outweigh throughput gains, resulting in slower convergence. This finding provides quantitative guidance for optimizing compute allocation in RLHF or RLAIF pipelines.
OpenAI revealed initial results for its custom AI inference chip, Jalapeño, claiming industry-leading speed and efficiency. The announcement highlights qualitative improvements in throughput and latency but lacks specific benchmark data or architectural details.
Hugging Face published a technical blog deriving continuous batching from first principles, starting with attention mechanisms and KV caching to optimize for throughput. The article helps developers understand the underlying logic of dynamic request scheduling and GPU utilization in LLM inference serving.
Hugging Face and Intel benchmarked language model inference on Google Cloud's 5th Gen Xeon (C4) versus previous-gen (N2). Results show C4 delivers 10-24x higher throughput for text embeddings and 2.3-3.6x for generation; factoring in price, C4 offers a 7-19x TCO advantage for embeddings. This supports the feasibility of deploying lightweight agentic AI workloads entirely on CPUs.