READ THE COMPLETE BOOK ONLINE
Inside DeepInfer
How an LLM Inference Engine Works
An answer appears one token at a time. Behind it, an inference engine decides which requests run next, what they can keep in memory and when that memory is safe to reuse.
Follow those decisions through DeepSeek’s DeepInfer, the engine behind its large-scale production services. The book takes you from a small generation loop to a service spread across many devices. Runnable experiments, source excerpts and diagrams connect each idea to the code that implements it.
Browse the contents ↓
From a question to the code
Trace a pause in generation to its scheduling decisions. Follow a cache address to its owner. Read a throughput curve through its counting assumptions, and a passing test through the claim it actually checks.
You need familiarity with functions, loops, arrays and elementary probability. The book introduces inference concepts as they become useful. Its 28 experiments run on a CPU; all but one use only Python’s standard library. The full engine is studied through its source.
Contents
Read in order, or use the repository map to find a feature in the code and its explanation in the book.
Start here
Chapters
- 01A generation loop
- 02Where the time goes
- 03The model shapes the engine
- 04State needs an owner
- 05Choose the next batch
- 06Keep the host and device busy
- 07Put experts on several devices
- 08Spend verification work wisely
- 09Reconstruct state approximately
- 10Different correctness contracts
- 11From an engine to a service