SiliconFlow's application for a Hong Kong stock exchange listing has been approved by the CSRC. As an independent ecosystem token provider, it unifies heterogeneous computing power to serve major LLMs, boasting over 10 million registered users and raising nearly 2.9 billion yuan in equity financing this year.
Inference Engine
推理引擎
Software system that loads models and executes forward passes to generate outputs
Curated Intelligence
DeepSeek unveiled DeepInfer at PyTorch Conf, an open-source inference engine claiming 155T tokens/day per cluster. It aims to challenge closed-source infrastructure dominance by offering a high-throughput, low-latency SOTA alternative.
DeepSeek has open-sourced its inference engine, DeepInfer, and disclosed operational metrics showing a single cluster processes 155 trillion tokens daily at $38 revenue per GPU-hour. The engine supports both NVIDIA Blackwell and Huawei Ascend chips, implying an estimated gross margin of ~82% based on B300 rental costs.
DeepSeek disclosed operational metrics for its inference engine, DeepInfer, revealing a single cluster serves 155 trillion tokens daily with $38 revenue per GPU-hour during off-peak times. Assuming B300 rental costs, this implies an ~82% gross margin, while the engine supports both NVIDIA Blackwell and Huawei Ascend hardware.
Startup Maxwell introduces a new compute paradigm sitting between traditional code execution and specialized hardware, aiming to make code 'think'. This approach challenges the separation of software and hardware data flows in current LLM inference engines like vLLM, arguing it is unscalable and suboptimal. Early access is now open.
Developer Niko1221 has open-sourced the Strata inference engine, enabling quantized Qwen3.8-Flash-Next (125B parameters) to run on consumer GPUs with 12GB+ VRAM, such as the RTX 5070. The solution achieves a generation speed of 94 tokens per second, demonstrating efficient deployment of large models on low-memory hardware.
Hugging Face announced that the vLLM backend in its transformers library now matches or exceeds the speed of custom vLLM implementations. Model authors can achieve ultra-fast inference without writing any optimization code by leveraging existing transformers implementations. This update significantly lowers the technical barrier for high-performance inference deployment.
SGLang announces integration of the Hugging Face Transformers backend. This update aims to improve model compatibility, though specific performance metrics and implementation details are not disclosed in the title.
TNG Technology Consulting published the first article in a series analyzing request queueing mechanisms in inference engines like vLLM and TGI. It compares scheduling strategies to optimize LLM serving performance under GPU resource contention in multi-user scenarios.
Intel introduces a deployment solution combining Optimum-Intel and OpenVINO GenAI. This toolchain aims to simplify LLM inference acceleration on Intel hardware by providing standardized integration paths.