News
What Is DeepInfer? DeepSeek’s Open-Source Inference Engine, Explained
DeepInfer is the inference engine DeepSeek uses to run its own AI models, open-sourced October 20, 2026. What it does, what is missing, and who it is for.
DeepInfer is the software DeepSeek uses to run its AI models for millions of people, and on October 20, 2026, the Chinese lab gave it away. The story shot to the top of Hacker News within hours, and the GitHub page passed 10,000 stars the same day.
If you have never heard the word "inference engine," that is fine. This page explains what DeepInfer is, what it can and cannot do, whether it is free, and why a release aimed at engineers still matters to people who just use AI apps.
What is DeepInfer?
An AI model is a huge file of numbers. On its own, it does nothing. To get an answer out of it, you need software that loads the model onto chips, feeds in your words, and produces the reply one token at a time. That software is called an inference engine. Training is the cooking. Inference is serving the meal, again and again, to everyone who orders.
DeepInfer is DeepSeek’s version. The first line of its GitHub page calls it "the high-performance inference framework that powers DeepSeek’s large-scale production services." In plain words: this is the same engine behind DeepSeek’s own API, now public at github.com/deepseek-ai/DeepInfer.
DeepSeek says it built DeepInfer mainly on top of vLLM, a popular open-source engine, with ideas taken from SGLang and Nvidia’s TensorRT-LLM. According to The Next Web, the vLLM team has already said it will borrow ideas from DeepInfer to improve its own support for DeepSeek’s model.
DeepInfer at a glance
| Question | Answer |
|---|---|
| Who made it? | DeepSeek, the Chinese AI lab |
| Released | October 20, 2026, announced on X, code on GitHub |
| License | Apache 2.0 (free, including for business use) |
| Models supported | DeepSeek V4.1 Flash only, for now |
| Hardware | Nvidia SM100 GPUs and Huawei Ascend 950 NPUs |
| What it does | Offline inference: running batches of requests through the model |
| Left out | KV cache system, full prefill/decode split, load balancing across replicas |
| Companion tool | deepseek-recipe, which turns text into tokens and back the same way DeepSeek’s official API does |
What can DeepInfer do?
The feature list is written for engineers, so here is what each part means in normal words:
- Splitting work across many chips. Big models do not fit on one chip. DeepInfer spreads the work out in several ways at once, including "expert parallelism," which places different expert parts of the model on different chips.
- Draft-token decoding. A fast helper guesses several words ahead, and the main model checks the guesses in one step. When the guesses are right, answers come out faster. This trick is often called speculative decoding.
- Structured outputs. It can force answers into a fixed shape, like valid JSON. That matters for AI agents, which need clean, predictable data to call tools.
- Repeatable results. On Nvidia chips, it supports deterministic sampling, so the same request can give the same answer every time. Developers love this for testing.
To try it, you need Linux, Python 3.12, and either CUDA 13 (Nvidia) or CANN 9.2 (Huawei). DeepSeek’s README also says its speed charts were measured on 4 SM100 GPUs for reading prompts and 64 for writing answers. That gives you a sense of the scale this was built for: data centers, not laptops.
What is missing from DeepInfer?
The release is narrower than "DeepSeek open-sources everything" headlines suggest. The README says plainly that this release provides offline inference. There is no ready-made API server. DeepSeek left out three parts you would need to run a real public service:
- a KV cache system, which remembers earlier parts of a conversation so the model does not reread them every time,
- full prefill/decode disaggregation, meaning separate machines for reading the prompt and writing the answer,
- load balancing across copies of the model.
DeepSeek says those pieces are tightly tied to its internal systems and are not fully open-sourced yet. Its performance figures are also its own. As The Next Web noted, citing Bloomberg, nobody outside the company has tested them independently yet.
DeepInfer vs DeepInfra: not the same thing
Expect some mix-ups. DeepInfra is a separate company that hosts many open models, including DeepSeek’s, and sells access through its API. DeepInfer is DeepSeek’s own engine software. One letter apart, very different things. A host like DeepInfra could, in theory, use ideas from DeepInfer to run DeepSeek models more cheaply.
Why did DeepSeek release it?
Three reasons stand out in the coverage:
- Cheaper DeepSeek everywhere. Cloud hosts and companies that run DeepSeek models themselves can copy its tricks and cut costs. More cheap hosts means more people using its models.
- Huawei gets equal billing. The engine treats Huawei’s Ascend chips as a first-class option next to Nvidia. Earlier in October, DeepSeek also published Huawei chip tools as an alternative to Nvidia’s CUDA.
- Timing. The release came right after the US Department of Homeland Security made public an order telling civilian federal agencies to remove DeepSeek products from their systems, as reported by CNBC and The Next Web. The order does not stop regular people or private businesses from using DeepSeek. Open code is one way to show what the software does.
If you want to know more about the model DeepInfer runs, start with What is DeepSeek V4.1 Flash?
What this means for you
You will probably never install DeepInfer. It needs racks of data center chips, and it only runs one model. But you will feel it. When the best tricks for running models become free to copy, the cost of every AI answer falls. That shows up as faster replies, cheaper plans, and more free features in the apps you already use.
There is a bigger point too. The work of AI is moving from "answer my question" to "do this task for me," which is what agentic AI means. Agents make many model calls for one job, so cheaper inference is exactly what makes them practical.
That is where our app fits. Agentic AI gives you a team of ready-made expert agents on iPhone and iPad. You hand a task to one agent, or put up to 4 on it together, and get back finished work. There is no account to make and you get 3 free tasks to try it. You never have to think about engines, chips, or caches.
Quick answers
What is DeepInfer? DeepInfer is the inference engine DeepSeek uses to run its AI models in production. DeepSeek open-sourced it on GitHub on October 20, 2026.
Is DeepInfer free? Yes. The code is under the Apache 2.0 license, so anyone can use and change it, including for business. The hardware to run it is the expensive part.
What models does DeepInfer support? Only DeepSeek V4.1 Flash at launch. DeepSeek has not said which models come next.
Does DeepInfer run on Huawei chips? Yes. It supports Huawei Ascend 950 NPUs as well as Nvidia SM100 GPUs.
Is DeepInfer the same as DeepInfra? No. DeepInfra is a separate company that hosts AI models. DeepInfer is DeepSeek’s own engine software.
Can I run DeepInfer on my computer? Not realistically. It needs Linux and data center chips. To use AI agents on your phone instead, see our guide to AI agents on iPhone.
Sources: DeepSeek: DeepInfer on GitHub · The Next Web · The New Stack · Crypto Briefing · Quartz · CNBC
