DeepSeek Open-Sources Inference Framework DeepInfer for V4.1-Flash

DeepSeek Open-Sources Its In-House Inference Framework DeepInfer: Apache 2.0 Licensed for Commercial Use, Currently Supports Only V4.1-Flash, Requires Data Center GPUs Like Blackwell or Ascend 950

DeepSeek has open-sourced DeepInfer, the inference framework powering its online services, along with the companion component deepseek-recipe.

Model files are merely a collection of trained parameters, akin to a recipe book. The actual process of handling requests, queuing them, and computing answers on GPUs relies on the inference framework, which functions like a kitchen's order dispatch system. An efficiently scheduled dispatch system allows the same GPUs to handle several times more requests while reducing wait times for each user.

DeepSeek states that DeepInfer is the infrastructure supporting large-scale serving of V4.1-Flash. The goal of open-sourcing it is to help the community save computational resources and reduce energy consumption.

Its primary acceleration technique is speculative decoding. Large models typically generate text token by token, requiring a full model pass for each new token. Speculative decoding employs a lightweight "draft" model to predict the next few tokens (up to 5 by default). The main model then verifies these predictions in a single pass; correct guesses are accepted immediately, while incorrect ones are regenerated.

End users will not directly interact with this framework. It requires NVIDIA Blackwell-generation data center GPUs (codenamed SM100) or Huawei Ascend 950 chips. Official performance benchmarks utilized between 4 and 64 such cards, making it unfeasible for personal computers. Currently, it only supports the V4.1-Flash model, though DeepSeek plans to include inference code for future models.

The primary beneficiaries are enterprises and cloud service providers aiming to self-host DeepSeek models, who can directly adopt DeepSeek’s optimized solutions. However, this release covers only "offline inference." Components such as context caching systems, disaggregated deployment architectures (separating input reading and output writing across different machines), and multi-node traffic distribution are not included. Building a complete online service still requires implementing these missing pieces.

deepseek-recipe addresses a different challenge: ensuring that self-hosted DeepSeek deployments align with official API behaviors regarding details like prompt construction and tool call parsing.

Acting like a front-end request handler, deepseek-recipe translates common request formats from OpenAI and Anthropic APIs into inputs readable by DeepSeek models. It also translates model outputs—such as reasoning processes and tool calls—back into standard formats, ensuring logic consistency with the official API.

DeepInfer is licensed under Apache 2.0, permitting free commercial use and modification. deepseek-recipe uses the MIT license, also allowing commercial use, and offers both Python and Rust implementations supporting V4 and V4.1 models.

DeepInfer: https://t.co/Tn3BtmA9Ec deepseek-recipe: https://t.co/1YArQ0vtYS

↳ Quote @deepseek_ai (DeepSeek): 🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

🔗