Preface

You type a question into ChatGPT and wait. An answer begins to appear. You ask a follow-up, referring to something in the reply, and the conversation continues. The text box gives little reason to think about the machinery serving it.

Behind that exchange, a service must keep many requests moving on shared hardware. Each answer needs repeated model calculations; each conversation brings a history that those calculations may have to read. Can one pass over the model’s weights serve several requests? How much work on the history can be saved before memory runs out? Once an answer is complete, when is its memory safe to reuse?

An inference engine organises the model computation, saved state and output generation behind such a service. This book studies those decisions in DeepSeek’s DeepInfer. It starts device work before the host program knows the previous result. It reserves space for guessed tokens that may be rejected. With a further optimisation enabled, it can shorten part of a prompt’s computation and reconstruct the missing state approximately. Each choice raises a question about what the next operation is allowed to assume.

We will follow these choices into DeepInfer’s code, using small experiments to expose what each operation needs from the last.

Why DeepInfer

DeepSeek identifies DeepInfer as the engine behind its large-scale production services. It has released an offline core alongside a deployment Q&A that describes the larger system: hardware, separate processing stages, routing and cache storage [1, 2]. The code shows how an engine advances a request; the Q&A describes how many such engines share the work and reach saved state. Together, they let us follow a request beyond the boundary of one process.

The model gives us another reason to look closely. Its attention design changes what a cache row contains. Sharing representations between layers changes which work can be omitted. Routing tokens to different experts changes which weights must be available on each device. These properties reach into memory allocation, scheduling and execution. Reading the model and engine together makes their constraints easier to understand.

DeepInfer was originally built primarily on vLLM. Its deployment Q&A credits SGLang and NanoFlow when describing asynchronous scheduling [1, 2]. If you know those systems, familiar names will help you find your bearings. We then ask what those names mean here: which design is retained, which behaviour differs, and which requirement comes from DeepSeek’s model. Fixed-version comparisons and the authors’ own acknowledgements give us evidence for those relationships; a familiar class name alone does not.

What you will be able to follow

By the end, you should be able to investigate a pause in generation by locating the work and waits that could explain it. You should be able to follow a cache address to its owner and determine when that storage can be reused. A throughput curve should lead you back to its counting assumptions. A passing test should lead you to the values it compares and the claim its assertion actually checks. Together, these skills give you a working understanding of inference engines and a route through DeepInfer’s architecture when you want to read further or change something.

Throughout, we distinguish teaching experiments, inspected implementation, documented production architecture and proposals made in this book. We also keep different claims of correctness separate. Equal values, values within a tolerance, samples from an intended distribution and useful answers each call for a different check. These distinctions matter whenever an optimisation changes the order, amount or location of computation.

You need familiarity with functions, loops and arrays, and enough Python to read a short program. Elementary probability will help with sampling. We introduce attention, device memory and distributed execution as the questions require them. Each technical chapter begins with an experiment whose result you can predict, then opens a connected path through the source. The excerpts are printed beside the explanation, with diagrams and concrete cases to keep track of the objects crossing each boundary.

What you can try

The small Python experiments let you make mistakes cheaply. Release a simulated memory block before its queued copy finishes, and another request’s data appears in the copy. Move expert work between simulated devices, and check that the answer survives the move. Enumerate a sampling rule’s possible outcomes, and see exactly where its output distribution changes. You can alter a condition, predict the consequence and check it before returning to the engine code.

There are twenty-eight experiments. They require Python 3.10 or later; all but one use only the standard library. The exception runs DeepInfer’s placement policy with PyTorch on a CPU. The experiments, their saved outputs and the source checks are available at https://github.com/jzstark/inside-deepinfer-code; Appendix C explains how the outputs were recorded. Assigned costs and simulated clocks expose dependencies and trade-offs; they do not measure accelerator performance.

The inspected engine release requires Python 3.12 and supports NVIDIA SM100 GPUs with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2 [1]. The Q&A recommends 64 prefill GPUs plus 64 decode GPUs as the smallest efficient service deployment, rather than a minimum for every offline experiment [2]. Most readers will therefore study the full engine through source walkthroughs.

The route through the book

We begin with one generation loop, then ask what batching saves and what a model must remember. Once that state has a size and an owner, we can decide which requests fit, which should run next and how host and device work can overlap. Experts spread the computation across devices; speculative decoding adds a choice about how much work is worth checking. Approximate reconstruction then tests the limits of what cached state can be recovered. The final chapters examine the guarantees these choices must meet and the service that connects engines to one another and to storage. A closing request trace brings the mechanisms back together.

The appendices support closer reading. Appendix A follows compression and indexing into attention. Appendix B maps README features to source entry points, traces a step’s calls and returns, and shows how processes and devices are grouped. Use it whenever a chapter enters an unfamiliar part of the repository. Appendix C identifies the source edition and explains reproduction; a terms table locates definitions.

The detailed walkthroughs cover the main runtime mechanisms and selected model-driven state paths. Low-precision storage is included; quantisation algorithms and general kernel tuning are outside the scope. Ascend kernel internals, vision architecture, checkpoint loading and the complete distributed initialisation and recovery paths are not covered systematically. Hyper-connections and Engram receive navigation entries rather than full derivations. The repository map marks these limits and supplies places to continue reading.

For now, start with a list of integers. One of them has just been generated, but the prediction function has not processed it yet. Keeping track of that small distinction will take us a long way into DeepInfer.