In one sentence: DeepInfer is DeepSeek's Apache-2.0 inference engine for DeepSeek-V4.1-Flash only — offline, B200/GB200 (SM100) or Ascend 950 only, not on H100/H200, and missing the KV cache, P/D split and load balancing a full service needs.
On 20 October 2026 DeepSeek published the engine behind its own API. This page separates what shipped from what the launch coverage implied — and what it means if you pay per token or run your own GPUs.
DeepInfer is DeepSeek's production inference framework for DeepSeek-V4.1-Flash, open-sourced under Apache 2.0 on 20 October 2026. It wires the kernel libraries DeepSeek had already published into one engine and adds Wide-EP decoding, DSpark speculative decoding, SWA bounded replay and deterministic execution modes. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs — not on H100/H200, and not on any other model.
Four kinds of reader, four different answers. Find your row first; the detail comes after.
No action. The official price list did not move with the release.
Stay on vLLM (0.31.0+) or SGLang. DeepInfer will not run on Hopper or AMD cards.
Worth benchmarking against your vLLM baseline — budget engineering time for the cache and routing layers.
The only production-derived open engine on that path, with the same three pieces missing.
At least two unrelated projects use this name, plus a similarly spelled company, and search results blend them together.
The inference framework on this page. Apache 2.0, V4.1-Flash only, SM100 and Ascend 950 only.
An older deployment toolkit and model registry for medical image analysis, maintained by researchers at Harvard Medical School's Surgical Planning Laboratory, Brigham and Women's Hospital and the University of British Columbia. No connection to DeepSeek.
A different company offering hosted inference, not an engine. Search results often blend the two spellings.
Six capabilities, each traceable to a line in the repository README. None of them is a hosted service.
FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels had been released separately. DeepInfer is the engine that connects them into one runtime.
Attention runs data-parallel, the MoE experts run expert-parallel. DeepSeek's production decode layout spreads one instance across 64–128 GPUs and rebalances hot experts with redundant copies.
A draft head proposes five tokens at a time and a confidence head ranks them. At the production operating point that cuts verified draft tokens by about 30% for a 4–7% throughput gain.
Instead of rebuilding the sliding-window cache layer by layer, the engine replays only the last 128 tokens of a prompt through the decoder. DeepSeek says this nearly halves total prefill compute — the cache is approximate by design.
The CPU prepares batch N+1 while the GPU runs batch N. Batch-invariant inference and deterministic sampling make runs reproducible — on NVIDIA hardware only.
Regex, JSON Schema and EBNF constraints are enforced during decoding. The companion deepseek-recipe library converts Chat Completions, Responses and Messages API requests with the same logic as DeepSeek's official API.
The README is unusually direct: a complete inference service also needs a KV cache system, prefill/decode disaggregation and load balancing. All three are tightly coupled to DeepSeek's internal infrastructure, so none of them is in this release.
By the account in DeepSeek's own long-form write-up on Zhihu: the deployment keeps a 3FS-backed cache with roughly 100 PiB of SSD, a 3-day TTL for agent traffic and 7 days for chatbot traffic. The design lives in that post; the code is not in the repository.
Most of the engine-side work is implemented, but splitting the three stages across machines is left to the operator. The smallest layout DeepSeek calls efficient is 64 prefill plus 64 decode GPUs.
The prefill, decode, vision-encoder and expert-parallel balancers are described in the deployment Q&A; their implementation is not part of the release.
Start with the card. Not every 2026 NVIDIA GPU can run this engine.
If you have H100, H200, H800 or A100: DeepInfer will not run on your hardware.
| Supported accelerators | NVIDIA SM100 (B200, GB200) · Huawei Ascend 950 |
| Not supported | Hopper (H100 / H200 / H800 / A100) and older NVIDIA architectures |
| OS and toolchain | Linux; GCC 13+; Python 3.12; CUDA 13 (NVIDIA) or CANN 9.2 (Ascend) |
The three numbers quoted most often in launch coverage each come with a different scope.
The published prefill chart uses four GPUs; the decode chart uses 64. Each stays within one scale-up domain.
Served by one DeepSeek production cluster — not a figure an open-source install reproduces.
Revenue at valley pricing — not a cost, and not a rental price you can shop for.
The two Pareto charts measure prefill throughput against latency and decode throughput against per-user speed. Both were produced by DeepSeek on NVIDIA hardware; as of 23 October 2026 we found no third-party reproduction. They are not end-to-end latency, not cost per token, and not a promise about your workload.
The repository also ships an evaluation harness. DeepSeek's reference Pass@1 figures for V4.1-Flash through that harness are AIME 2026 99.9%, GPQA Diamond 93.4% and HLE-Full 42.6% — those measure the model, not the engine.
For most teams the practical question is not which engine is fastest, but which one runs on the hardware they already have.
| DeepInfer | vLLM | SGLang | |
|---|---|---|---|
| Model scope | DeepSeek-V4.1-Flash only | General-purpose, many architectures | General-purpose, many architectures |
| Hardware | SM100 (B200/GB200), Ascend 950 | H200, GB300, AMD MI3xx and more | GB300 and other NVIDIA platforms |
| Serving stack | Most P/D engine work is in place, but the scheduler, cache service and load balancer are not — you wire the split yourself | OpenAI-compatible server, P/D disaggregation, KV offload | Server, HiCache, P/D disaggregation |
| V4.1-Flash path | Loads DeepSeek-V4.1-Flash-1017 by default | vLLM 0.31.0 or newer | Day-one support |
| Best fit | Reproducing DeepSeek's serving path on SM100 or Ascend | Broad hardware, many models | High-throughput online serving for V4.1 |
In one line: DeepInfer is in play only when your cards are SM100 or Ascend 950 and you are willing to build the serving layers yourself; otherwise vLLM or SGLang is the default answer. DeepInfer began life as a vLLM-based fork, and its README credits vLLM, SGLang and TensorRT-LLM as inspirations.
The short version: nothing on your invoice.
| deepseek-flash (per 1M tokens) | Off-peak | Peak |
|---|---|---|
| Input · cache hit | $0.003 | $0.006 |
| Input · cache miss | $0.15 | $0.30 |
| Output | $0.60 | $1.20 |
Open-sourcing an engine does not reprice an API. The half of the arithmetic that launch coverage skipped is the self-hosting side: the smallest deployment DeepSeek calls efficient is 128 GPUs, before the KV cache storage cluster and the routing layer you have to write. If you are not already at that scale, the API remains the cheaper path — and the $38 per GPU-hour figure is revenue, not a rental price you can shop for.
Unless you already operate cluster-scale SM100 or Ascend 950 capacity, treat DeepInfer as a reference architecture rather than a deployment option. It is worth reading; it is not worth buying GPUs for.
The official path, straight from the README, with nothing wrapped around it.
First create and activate an empty Python 3.12 environment, then run the three steps below.
export DEEPINFER_ACCELERATOR=nvidia # use "ascend" for Ascend; keep it exported
./build.shpython examples/offline_inference.pyPer vLLM's recipe, the FP8 checkpoint is roughly 511 GB on disk. Plan storage before you plan throughput.
DeepInfer is DeepSeek's open-source inference framework for DeepSeek-V4.1-Flash, released on 20 October 2026 under the Apache 2.0 licence. It packages the kernel stack DeepSeek had already published (FlashMLA, DeepGEMM, DeepEP and others) into one engine with expert parallelism, DSpark speculative decoding, SWA bounded replay, asynchronous scheduling and deterministic modes. It is an offline engine, not a hosted API.
No. DeepInfer serves exactly one model, DeepSeek-V4.1-Flash, and only on NVIDIA SM100 or Huawei Ascend 950 hardware. What it omits are the parts of a complete online service around that engine: a KV cache system, the load balancer across data-parallel replicas, and the orchestration around P/D disaggregation — the disaggregation itself is largely implemented, the scheduler and wiring are not. vLLM and SGLang remain the practical choice for other models, other GPUs and OpenAI-compatible serving.
NVIDIA SM100 (Blackwell) GPUs — B200 and GB200 systems — and Huawei Ascend 950 NPUs. The README lists only those two accelerator families, so Hopper and older NVIDIA architectures are outside the supported set. It does not say whether batch-invariant inference and deterministic sampling are available on Ascend.
No. Hopper-generation cards are not on the supported list. For V4.1-Flash on H200, vLLM's recipe (0.31.0 or newer) and SGLang's day-one support are the documented paths. DeepInfer targets DeepSeek's own SM100 and Ascend 950 deployment.
Not directly. The deepseek-flash list price did not change with the release: $0.15 input and $0.60 output per million tokens off-peak, $0.30 and $1.20 at peak, and $0.003 (off-peak) or $0.006 (peak) for cache hits. Self-hosting means a 64 prefill plus 64 decode reference deployment — 128 GPUs, DeepSeek's smallest efficient unit — and then building the KV cache and routing layers the release does not include.
Page data was read from the primary sources below on 23 October 2026. Hardware support, prices and benchmarks change — check the official repository and docs before you deploy.