References

[1]   DeepSeek-AI. DeepInfer: release overview. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md. Local file: README.md. Inspected 20 October 2026.

[2]   DeepSeek-AI. DeepSeek V4.1 Flash Inference Deployment Q&A. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md. Inspected 20 October 2026.

[3]   Jzstark. The Economist Writing Style: A Practical Reference. https://github.com/jzstark/eco-writing-style/blob/main/eco-style.md. Read 20 October 2026. The project’s chosen editorial reference.

[4]   DeepSeek-AI. Offline LLM entry point. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/entrypoints/llm.py. Symbols: LLM.generate, LLM._add_request, LLM._run_engine. Offline preemption default: line 68.

[5]   DeepSeek-AI. LLM engine. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/llm_engine.py. YOCO prompt-logprob restriction: lines 528–536. Engine methods: add_request, step, _execute_model_async_step and post_process. Lifetime helper: DeferredRelease, lines 67–116. finalize_seq: lines 804–806; pre_sampler_sync: lines 830–846; metadata collection in post_process: line 1013.

[6]   DeepSeek-AI. Sequence state. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/sequence.hpp. Symbols: SequenceData, SequenceStatus.

[7]   DeepSeek-AI. Worker execution. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/worker.py. Symbols: Worker.execute_model_step, Worker._post_process_step, Worker.flush_inflight_reqs. Step order: lines 113–189; _correct_seqlens_qk: lines 191–245; event wait and graph-timer flush: lines 262–266. Local source hashes for these code references are recorded in source-manifest.json; the source links above pin the verified commit.

[8]   DeepSeek-AI. C++ scheduler. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/scheduler.hpp. Symbols: Scheduler::add_sequence, lines 135–142; Scheduler::schedule, lines 667–736; Scheduler::_preempt, lines 766–790.

[9]   DeepSeek-AI. Request output storage. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/outputs.hpp. Symbols: RequestOutputPool::allocate, RequestOutputPool::release, lines 64–80.

[10]   DeepSeek-AI. Request output Python binding. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/pybind_output.cpp. RequestOutput.clone implementation: lines 16–18; declaration: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/_C/engine.pyi#L182.

[11]   vLLM contributors. Sequence definitions, v0.6.3. https://github.com/vllm-project/vllm/blob/fd47e57f4b0d5f7920903490bce13bc9e49d8dba/vllm/sequence.py#L625. Symbol: SequenceGroup. A historical comparison, not an identified fork point.

[12]   vLLM contributors. LLM engine, v0.6.3. https://github.com/vllm-project/vllm/blob/fd47e57f4b0d5f7920903490bce13bc9e49d8dba/vllm/engine/llm_engine.py#L1284. Symbol: LLMEngine.step; its docstring describes freeing sequence groups.

[13]   vLLM contributors. Scheduler, v0.6.3. https://github.com/vllm-project/vllm/blob/fd47e57f4b0d5f7920903490bce13bc9e49d8dba/vllm/core/scheduler.py. Queues: lines 340–348; preemption selection: lines 1472–1499; recomputation: lines 1501–1510; swap-out: lines 1529–1543. Compared with DeepInfer’s pinned scheduler, not an identified ancestor revision.

[14]   DeepSeek-AI. Main-model FLOPs estimator. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/flops_estimator.hpp. Symbols: flops_coef, FlopsEstimator::calc_flops, lines 13–45. Replay query ranges and cost terms: lines 237–320.

[15]   DeepSeek-AI. Offline benchmark. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/benchmark.py. Symbols: measure_decode, run_decode_step; endpoint storage, lines 380–381; measurement and profile windows, lines 499–519; token-accounting comments, lines 405, 407 and 539; topology and rank-0 reporting, lines 71–76 and 521–560; YOCO mask-mode option, lines 45–50.

[16]   DeepSeek-AI. Data-parallel process launcher. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/_launch.py. Symbols: launch_dp, run_dp.

[17]   DeepSeek-AI. Example engine configuration. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/_config.py. Symbol: setup_cuda_visible_devices, lines 55–64. Example chunked-prefill setting and caller overrides: lines 118–123.

[18]   Samuel Williams, Andrew Waterman and David Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009, pp. 65–76. https://doi.org/10.1145/1498765.1498785.

[19]   Ashish Vaswani et al. Attention Is All You Need. 2017. Section 3.2.1: scaled dot-product attention. https://arxiv.org/abs/1706.03762.

[20]   DeepSeek-AI. DeepSeek model execution. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/deepseek.py. Symbol: DeepseekModel.forward, lines 154–248. Wrapper sampling handoff: DeepseekForCausalLM.sample, lines 383–401. YOCO compatibility checks: lines 117–138.

[21]   DeepSeek-AI. Decoder layer. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/decoder_layer.py. V4 attention construction: line 85; attention call: lines 155–161. Feed-forward call: lines 169–173. Full/compact boundary normalisation: lines 134–152.

[22]   DeepSeek-AI. V4 attention runtime. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/attn.py. Class: V4Attention. Query and SWA projection: lines 386–464; decode attention: lines 628–670; forward path: lines 752–880. Full-source block KV: _store_yoco_full_kv, lines 672–721.

[23]   DeepSeek-AI. Cache engine and state layout. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache_engine.py. Format selection: lines 46–56; ring sizes: lines 88–110; stored/dropped classification: lines 113–116; SWA byte budgets: lines 261–280; cache view: lines 407–422; state-slot count: lines 201–202; cache fields: lines 206–219; StateBlock: lines 221–241 and 306; select_state_slice_to_store: lines 763–817; load_state_from_store_to_prefill_cache: lines 821–848.

[24]   DeepSeek-AI. KV-cache formats. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cache_common.py. Symbols: KVCacheFormat, V41_FORMAT, lines 24–137, including main FP8/FP4 and index FP4 formats.

[25]   DeepSeek-AI. Attention metadata. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/input_metadata/input_metadata.py. V4 attention information: lines 343–385. CUDA decode address construction: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/prepare_decode_metadata.py#L138.

[26]   DeepSeek-AI. KV index expansion. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/get_kv_index.py. Decode SWA wrapper: lines 97–112. CUDA ring branch: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/get_kv_index.py#L59.

[27]   DeepSeek-AI. Attention backend. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/attention_backend.py. Symbol: decode_mla_mqa_sparse, lines 77–146.

[28]   DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024. https://arxiv.org/html/2405.04434v5#S2.SS1.SSS2. Sections 2.1.2–2.1.3: joint KV compression and the separately cached position-encoding component.

[29]   DeepSeek-AI. MLA model projections. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/mla_attn.py. Low-rank projection container: lines 26–59; value and shared KV projections: lines 224–245. Runtime output projection: _wv_and_o_proj in reference [22], lines 191–225.

[30]   DeepSeek-AI. Environment defaults. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/environ.py. Draft limit and backtracking allowance: lines 129–132; extra state storage: lines 163–164; YOCO options: lines 188–194.

[31]   DeepSeek-AI. Verification query counts and decode dispatch. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp_varlen.hpp#L150. Fixed-length query count: lines 150–158. The scheduler in reference [8] marks decode as non-prompt at lines 651–659 and transfers that flag at line 707. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/model_runner.py#L120. Input-metadata construction: lines 120–146; CUDA query positions: reference [25], kernel lines 112–144.

[32]   Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023.

[33]   DeepSeek-AI. Scheduler memory ownership. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/scheduler.hpp. Joint checks and state cloning: lines 194–212; forked admission: lines 412–437 and 602–618; prompt allocation: lines 518–519; decode extension and preemption: lines 629–640; step key length: line 697; release: lines 738–745; _append_slot: lines 817–828; processed-count correction: lines 927 and 998–999; fork_seq: lines 1061–1122; block copies: lines 1288–1300.

[34]   DeepSeek-AI. Block and state-slot allocator. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/block_manager.hpp. Symbols: Blocks, lines 27–117; BlockManager::can_allocate_nolock, lines 128–136; BlockManagerMutexAcquire::allocate_seq, lines 207–214.

[35]   DeepSeek-AI. Scheduler and allocator Python bindings. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/pybind_scheduler.cpp. unpin_blocks bound to BlockManager::free_blocks: line 68.

[36]   DeepSeek-AI. Worker memory profiling. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/base.py. State reservation and block count: lines 237–246; cross-rank result: lines 264–267.

[37]   DeepSeek-AI. Scheduler admission, step selection and preemption. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/scheduler.hpp. Prefill cache watermark: line 80; completion-engine restriction: lines 102–104; step selection: lines 275–345; prefill counter reset: lines 675–680; prompt admission and chunking: lines 351–553; decode ordering: lines 624–659; metadata construction: lines 688–728; preemption: lines 766–790; stale-result comment and guard: lines 917–924; unfinished-chunk requeue: lines 1016–1019. State reset helpers: reference [6], lines 134–142, 223–227, 258–261 and 307–311.

[38]   DeepSeek-AI. V4 input preparation. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/input_metadata/prepare_tensor_input_v4.cpp. Prefill SWA write destinations: lines 98–109; YOCO visibility: lines 149–170; truncation metadata: lines 203–271; update_yoco_prompt_v4: lines 349–360.

[39]   DeepSeek-AI. Engine construction arguments. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/arg_utils.py. Preemption, engine-type and chunking defaults: lines 40–42.

[40]   DeepSeek-AI. ZOS input preparation. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/zos.cpp. Fresh-row tokens: lines 99–113; prepare_zos_inputs: lines 142–196; update_metadata_for_sampling_zos: lines 279–305.

[41]   DeepSeek-AI. Worker step state and ZOS metadata. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/zos_status.py. WorkerStepState: lines 9–20; MetadataZOS: lines 23–40.

[42]   DeepSeek-AI. ZOS token gathering. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/gather_input_ids_zos.py. gather_input_ids_zos contract: lines 10–32. Caller: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/input_metadata/prepare_tensor_input_zos.py, prepare_token_ids_zos, lines 181–244.

[43]   DeepSeek-AI. ZOS key-length correction. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/compute_seqlens_k_zos.py. Kernel wrapper and formula: lines 10–33; reference implementation: lines 35–52.

[44]   DeepSeek-AI. Engine types and ZOS default. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/engine_type.py. Default ZOS engine types: lines 16–17. Environment default applied in reference [39], lines 51–55.

[45]   DeepSeek-AI. CUDA graph runner. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cuda_graph_runner.py. Graph data contract: lines 26–38; batch-size buckets: lines 41–51; selection and replay: lines 143–154. Runner construction: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/model_runner.py, lines 72–79; YOCO keep-length assertion: lines 107–113.

[46]   DeepSeek-AI. Host and device timers. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/timer.py. CPUTimer.record: lines 50–63; CUDAGraphTimer.flush_out: lines 196–212.

[47]   DeepSeek-AI. V4 MoE execution. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/mlp.py. Fused-path condition: lines 184–188; slot counts: lines 196–202; gate: lines 359–401; routed experts: lines 406–459; dispatch: lines 461–491; combine: lines 493–511; shared expert: lines 513–526; fused path: lines 528–563; forward: lines 565–616. Communication SM limit: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/parallel_utils/userbuffer.py, lines 40–51.

[48]   DeepSeek-AI. Expert load balancer. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/eplb/__init__.py. Slot padding: lines 21–28; initial map: lines 31–39; state and window: lines 42–129; step: lines 131–165; _rearrange_experts: lines 222–288; per-layer views: lines 346–376. Step call site: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/deepseek.py, lines 402–404.

[49]   DeepSeek-AI. EPLB policy. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/eplb/policy.py. _batch_greedy_indices: lines 12–70; rebalance_experts_global: lines 73–140.

[50]   DeepSeek-AI. EPLB weight shuffle. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/eplb/shuffle.py. Point-to-point path: lines 65–147; intranode mirror path: lines 150–218.

[51]   DeepSeek-AI. Model loader. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/model_loader.py. Redundant-slot resolution: lines 83–107 and 122–123.

[52]   DeepSeek-AI. DeepGEMM SM100 mega gate. https://github.com/deepseek-ai/DeepGEMM/blob/1fc9c66975ee322d628b3288534ca0e9ce12a64d/deep_gemm/include/deep_gemm/epilogue/sm100_mega_gate.cuh. Replica choice: lines 204–206. Commit pinned by DeepInfer’s vendor/DeepGEMM submodule at the book’s source edition.

[53]   DeepSeek-AI. TileKernels 2.1.0 top-k gate. Package tile-kernels==2.1.0, required by DeepInfer’s requirements.txt, line 17. File tile_kernels/moe/moe_to pk_gate_forward_cuda.py: replica choice at lines 41 and 282–284. Package and file hashes are recorded in source-manifest.json.

[54]   DeepSeek-AI. Expert Parallelism Load Balancer, standalone repository, commit d52c72d. https://github.com/deepseek-ai/EPLB/blob/d52c72d5b2f2fb4c41afbf8eb21366820239913d/eplb.py. Scalar packing: lines 27–40; replica allocation: lines 58–71; entry point and topology selection: lines 131–162. The README describes load prediction as outside the repository’s scope: https://github.com/deepseek-ai/EPLB/blob/d52c72d5b2f2fb4c41afbf8eb21366820239913d/README.md.

[55]   DeepSeek-AI. DeepInfer dependency declarations, at the book’s fixed revision. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/.gitmodules; https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/requirements.txt. The gitlinks pin DeepGEMM and DeepEP; requirements pin TileKernels 2.1.0. Versions and inspected file hashes are recorded in source-manifest.json.

[56]    Yaniv Leviathan, Matan Kalman and Yossi Matias. Fast Inference from Transformers via Speculative Decoding. ICML, PMLR 202, 19274–19286, 2023. Algorithm 1 and Appendix A.1. https://proceedings.mlr.press/v202/leviathan23a.html.

[57]   Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper. Accelerating Large Language Model Decoding with Speculative Sampling. 2023. Algorithm 2 and the distribution-preservation argument. https://arxiv.org/abs/2302.01318.

[58]   DeepSeek-AI. Speculative verification and accepted-prefix collection. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/sampler.py, Sampler._verification, beginning at line 411. Sampler.forward, lines 114–144, selects hidden rows and computes logits before verification or ordinary sampling. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/fused_sample.py, acceptance: lines 378–399; residual: lines 435–445; fallback: lines 506–518. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/sampler.py, prepare_verification_metadata_tl, get_accept_result_tl and finalize_verification_tl.

[59]   DeepSeek-AI. Sampling transforms and proposal vocabulary alignment. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/sampler_utils.py, process_sampling, lines 113–197. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/sampler.py, remap_draft_topk.

[60]   DeepSeek-AI. Draft probability penalty and its ideal-arithmetic bound. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/common.py. Docstring: lines 3–21; DRAFT_PROB_PENALTY: line 53.

[61]   DeepSeek-AI. Fixed-point sampling helpers. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/random_sampling.py. Symbols: Config, float_to_fixed, accumulate, sample_offset, sample_select.

[62]   DeepSeek-AI. Variable verification budget and runtime estimates. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp_varlen.hpp. Estimators: lines 19–132; runtime update: lines 140–148; schedule: lines 150–267; confidence-round comment: lines 159–167. CPU prefix products: csrc/engine/scheduler.hpp, lines 1239–1250, at the same fixed revision (reference [37]).

[63]   DeepSeek-AI. Startup decode performance curve. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/model_runner.py, profile_decode_perf, lines 562–631. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/base.py, warm_up_model, lines 275–281. Feedback: deepinfer/engine/llm_engine.py, lines 995–1002, at the same revision (reference [5]).

[64]   DeepSeek-AI. DFlash proposal generation and confidence. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/mtp/dflash.py. Input embedding, draft forward and sampling: lines 230–345.

[65]   DeepSeek-AI. Stored MTP proposals. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/mtp_info_pool.py. MTPInfoPool: slot layout, reserved slot zero and store.

[66]   DeepSeek-AI. Device allocation of a fixed verification budget. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cuda/schedule_mtp_varlen.py. compute_seqlens_q_by_confidence_rates_tl, lines 14–83. Call and D2H event: deepinfer/worker/worker.py, lines 195–234; sampling metadata update: lines 146–152 (reference [7]).

[67]   DeepSeek-AI. Verification confidence metric. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/sampler.py, compute_confidence_error_ref, lines 584–589. CUDA accumulation: tilelang_ops/cuda/sampler.py, lines 368–374; probability proxy: tilelang_ops/cuda/fused_sample.py, lines 471–479 and 520–522 (reference [58]). https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/stats_logger.py, normalisation: lines 234–236; reported name: line 309.

[68]   Yutao Sun et al. You Only Cache Once: Decoder-Decoder Architectures for Language Models. 2024, arXiv:2405.05254v1, sections 2.2–2.3. https://arxiv.org/html/2405.05254v1#S2.SS2.

[69]   DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. 2026, arXiv:2609.19969v1, sections 2.2, 3.2.2 and 4.2.1. https://arxiv.org/html/2609.19969v1#S2.SS2.

[70]   DeepSeek-AI. Hugging Face configuration adapter. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/hugging_face/config.py. SWA shared-hidden default: line 47; segmented maps: lines 66–85; _n_prefill_layer: lines 158–161; window mapping: line 284.

[71]   DeepSeek-AI. YOCO runtime predicates and compaction. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/yoco.py. is_yoco_consumer_layer: lines 19–21; compact_yoco_boundary: lines 93–123.

[72]   DeepSeek-AI. Cross-layer sharing topology and runtime. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/cross_layer.py. Boundary validation: lines 114–116; CrossLayerRuntime: lines 277–325.

[73]   DeepSeek-AI. YOCO input metadata views. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/input_metadata/yoco.py. get_mask_view: lines 84–92; get_consumer_view: lines 94–128 for the single-context-rank path, with redistribution below.

[74]   DeepSeek-AI. FLOPs estimator reference checks. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/engine/test_flops_estimator.py. Fixture: lines 26–31; per-query reference: lines 34–145; mode, chunk and Engram cases: lines 148–178. Inspected, not run for this book.

[75]   DeepSeek-AI. YOCO mask/compact equivalence checks. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/yoco/README.md. Driver: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/yoco/yoco_equiv.py. Baseline selection: line 536; baseline validation: lines 793–797. Full-vocabulary guard: line 566; temperature/top-p: lines 674–675. Inspected, not run for this book.

[76]   DeepSeek-AI. Batch-invariance configuration and library boundaries. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/runtime_config.py, lines 54–69. Attention predicates and calls: deepinfer/model_executor/ models/v4_modules/attn.py, lines 87–92, 666 and 1002 (reference [22]). https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/parallel_utils/userbuffer.py, lines 72–80. FlashMLA imports and sparse-decode flag forwarding: deepinfer/model _executor/layers/attention_backend.py, lines 2 and 141; direct fused-operation import: deepinfer/model_executor/models/v4_modules/attn.py, line 14. The gitlink at vendor/FlashMLA pins 2e5429fc5653bab6e081f09477126f731882a6a9; .gitmodules identifies the DeepSeek repository. All caller evidence and the gitlink are from the same DeepInfer revision.

[77]   DeepSeek-AI. Released numerical model checks. Inspected, not run for this book. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/models/test_batch_invariant.py, result fields at lines 41–47 and golden replay at lines 96–105. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/models/test_ve_batch_invariant.py, embedding comparison at lines 113–127. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/models/test_kl.py, golden generation at lines 80–82; comparison at lines 177–209. Prompt sampling parameters: https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/prepare_sample.cpp, lines 186–188; prompt-logprob calculation: deepinfer/model_executor/ layers/sampler_utils.py, lines 52–78. Temperature, top-p/top-k and log-softmax reference: deepinfer/model_executor/layers/tilelan g_ops/fused_sample.py, lines 137–150 and 209–211 (reference [79]). https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/testing.py, cal_diff, lines 4–12. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/models/test_offline_inference.py, platform configuration and stored offline_decode_ppl.json comparison.

[78]   DeepSeek-AI. Golden overrides and verification metadata. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/sampler.py, reference metadata at lines 13–25 and override at lines 227–250. CUDA implementation: deepinfer/model_executor/layers/tilelang_ops/cuda/sampler .py, lines 252–276, at the same revision (reference [58]).

[79]   DeepSeek-AI. Sampling keys and evaluation defaults. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/fused_sample.py, MTP offset at line 15 and namespace selection at lines 93 and 208. CUDA keys: tilelang_ops/cuda/fused_sample.py, lines 227, 392, 450 and 481 (relative to deepinfer/model_executor/layers/; reference [58]). Packed positions: tilelan g_ops/sampler.py, lines 56–66, and tilelang_ops/common.py, line 52; expansion caller: sampler_utils.py, lines 162–163 (reference [59]). DFlash proposal positions: deepinfer/model_executor/models/mtp/dflash.py, lines 290–294 (reference [64]). https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/eval/run_eval.py, per_sample_seed and _setup_env, including lines 121–129.

[80]   DeepSeek-AI. Randomised PIT and sampler checks. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/models/test_sampler.py, settings at lines 39–45, statistical helpers at lines 50–94 (trajectory rejection filter at lines 85–88), and failure conditions at lines 242–250. Inspected, not run for this book. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/sum_smaller_probs.py, inclusive CDF definition and reference at lines 10–44.

[81]   DeepSeek-AI. Structured-output matcher state and sampling masks. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/structure_output/matcher.py, rollback capacity at line 20; token acceptance, masking and rollback at lines 134–235; failed futures at lines 297–338. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/sampler.py, matcher updates at lines 339–386 and masking at lines 433–437. Engine enqueue failure: deepinfer/engine/llm_engine.py, lines 621–622; scheduler failure/preemption: csrc/engine/scheduler.hpp, lines 769–788 and 973–980 (references [5, 37]). Trigger/bound interface: deepinfer/sampling_params.py, lines 39–43. XGrammar version: requirements.txt, line 21, all at the same revision.

[82]   DeepSeek-AI. Structured-output examples and format assertions. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/_structured_output.py, settings at lines 37–41; termination and format assertions at lines 42–63. Invoked by tests/models/test_offline_inference.py with check_structured_output=True. Inspected, not run for this book.

[83]   DeepSeek-AI. Engine roles and the offline measurement boundary. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/arg_utils.py, NORMAL default at line 41. Prompt-only assertion: d e e p i n f e r / e n g i n e / l l m _ e n g i n e . p y, lines 947–949, at the same revision. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/README.md, “Stage decoupling and metrics”. Benchmark construction: examples/benchmark.py, lines 173–188; example factory: examples/_config.py, lines 112–125, at the same revision.

[84]   DeepSeek-AI. State selection, restoration and layout tests. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache_engine.py, select_state_slice_to_store, lines 763–818; load_state_from_store_to_prefill_cache, lines 821–848. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/tests/worker/test_state_cache_ops.py, padding removal at lines 76–84; round-trip comparison at lines 188–215. Inspected, not executed for this book.

[85]   DeepSeek-AI. Performance sweeps, plotting code and published decode figure. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/examples/sweep/plot_pareto.py, record selection at lines 37–55; conversion and axes at lines 58–75; subtitle at lines 91–100; point connection at line 150. Sweep driver: examples/sweep/run_sweep.sh, at the same revision. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/images/decode_pareto.png. Chapter 11’s 246 tokens/s/user is an approximate visual reading at 64 requests per DP rank, not a recovered benchmark record.

[86]   DeepSeek-AI. 3FS and DeepSeek Recipe. https://github.com/deepseek-ai/3FS; https://github.com/deepseek-ai/deepseek-recipe. Public project boundaries checked on 22 October 2026. The DeepInfer deployment Q&A does not identify the deployed revisions; this book does not inspect their internal service implementations.

[87]   DeepSeek-AI. Runtime temporal compression. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/compress.py, KProject and Compressor, especially lines 74–167. Projection: deepinfer/model_e xecutor/layers/mla_attn.py, SoftmaxAPEGatedColumnParallelLora, at the same revision.

[88]   DeepSeek-AI. Compression interfaces, references and checks. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/compress.py, compress_and_update_state_prefill, compress_and_update_state_decode, reference formulas at lines 178–248, and device tests in the same file. Tests inspected, not run. Live state allocation: deepinfer/worker/cache_engine.py, lines 123–133, 441–465, at the same revision.

[89]   DeepSeek-AI. Compressed positions, store slots and visible lengths. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/input_metadata/prepare_tensor_input_v4.cpp, lines 46–91 and 591–601.

[90]   DeepSeek-AI. Index attention and candidate selection. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/index_attn.py, MQA, store_index_kv, compute_chunk_size, _topk_prefill, _topk_decode and candidate selection. Configuration: deepinfer/hugging_face/config.py, lines 111–134, at the same revision. Index scoring and top-k implementations in independent libraries are identified at the import/call boundary, not audited here.

[91]   DeepSeek-AI. Segmented sharing maps and runtime data. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/cross_layer.py, SharedLayerTopology, refinement checks at lines 181–189, and CrossLayerRuntime.

[92]   DeepSeek-AI. Paged index conversion. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/flash_mla_utils.py, convert_kv_index; CUDA implementation in deepinfer/model_executor/layers /tilelang_ops/cuda/flash_mla_utils.py, lines 31–37, at the same revision. The kernel and tensor reference use the output lane’s position relative to the valid count; they do not use a sentinel-value guard.

[93]   DeepSeek-AI. Main/index KV storage and sparse attention read. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/v4_modules/attn.py, main/index storage at lines 515–587; decode cache arguments and attention call at lines 640–669; full-layout YOCO storage at lines 672–705.

[94]   DeepSeek-AI. Engine drivers, workers and rank groups. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/base.py, WorkerBase rank setup and run_workers. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/parallel_utils/parallel_state.py, ParallelProcessGroup.create_process_groups. At the same revision: example s/_launch.py, run_dp; examples/_config.py, setup_cuda_visible_devices; deepinfer/config.py, ParallelConfig; deepinfer/engine/llm_engine.py, _init_workers; deepinfer/model_executor/layers/tilelang_ops/prepare_cp_m irror.py, lines 135–146; csrc/engine/scheduler.hpp, line 90.

[95]   DeepSeek-AI. Model registration, wrappers and navigation entries. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/__init__.py, _MODELS; deepinfer/model_executor/models/deepseek_vl.py, DeepseekVLForCausalLM.forward, at the same revision. Core model: deepinfer/model_executor/models/deepseek.py. Navigation only: deepinfer/model_executor/models/v4_modules/hc.py, HCLayerMixin, HCHead; v4_modules/engram.py, EngramRuntime; v4_modules/builder.py (the latter two paths relative to the models directory). These entries do not evaluate the model mechanisms or weight-loading implementation.

[96]   DeepSeek-AI. Native scheduler and output bindings. https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/pybind_scheduler.cpp, schedule and process_outputs bindings. At the same revision: csrc/engine /pybind_output.cpp, build_request_outputs; csrc/engine/outputs.hpp, RequestOutputPool and output construction.

[97]   DeepSeek-AI. Repository layout and platform dispatch. https://github.com/deepseek-ai/DeepInfer/tree/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01. Platform dispatch: deepinfer/model_executor/layers/tilelang_ops/ops.py; platform selection: deepinfer/platform.py; dependency repositories: .gitmodules. Directory entries in Appendix B are navigation, not a claim to have inspected all files in each directory.