DeepSeek Open-Sources Inference Framework DeepInfer for Ascend and NVIDIA

DeepSeek officially states that while the capability boundaries of large models are defined by the models themselves, providing ultra-large-scale model services at a reasonable cost depends on an intermediate layer that has long remained obscure to the public: the inference framework.

Training enables models to acquire knowledge, while inference makes them usable for people. Facing today’s massive volume of user calls, the inference framework is responsible for orchestrating and scheduling vast computing clusters. Its efficiency directly determines the computational costs and power consumption of AI services.

By sharing its internal deployment experiences, DeepSeek aims to help developers and enterprises optimize throughput and latency in large-scale inference, thereby efficiently building private and public cloud deployments, reducing computational costs, and lowering energy consumption. This effort seeks to make intelligence accessible to a broader audience at lower prices, achieving true democratization of large models.

Based on DeepInfer, DeepSeek has already implemented high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash. In the future, inference code and subsequent optimizations for new DeepSeek models will continue to be updated here.

• DeepInfer Inference Framework: Integrates high-performance operators with a deeply optimized inference engine, supporting capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output. It is compatible with both NVIDIA and Huawei Ascend platforms.
• deepseek-recipe Encoding/Decoding Components: Production-grade token encoding and decoding components released simultaneously as open source. These are fully aligned with the processing logic of the official DeepSeek API, ensuring that the community receives an API experience consistent with official deployments.

IT Home provides the following open-source addresses: