APPENDIX B

Repository and execution map

The chapters organise DeepInfer by mechanisms. This appendix organises the same implementation by processes, call boundaries and directories. Use it to choose a starting symbol, then return to the chapter that explains its behaviour. Line anchors refer to the pinned edition in Appendix C.

python3 labs/28_ranks.py

DP, CP and EP mean data, context and expert parallelism. The lab translates rank arithmetic; it starts no workers. DP4/CP1 gives four drivers, each using one device. DP2/CP2 gives two drivers, each with one spawned CP follower. EP4 groups those same four device workers in both configurations. For DP2/CP2 the computed memberships are:

DP groups: [[0, 2], [1, 3]]
CP groups: [[0, 1], [2, 3]]
EP groups: [[0, 1, 2, 3]]

The grouping arithmetic in the lab is small enough to inspect directly:

10def layout(dp, cp, ep):
11    world = dp * cp
12    assert world % ep == 0
13    workers = [(e * cp + c, e, c, "driver" if c == 0 else "child")
14               for e in range(dp) for c in range(cp)]
15    dp_groups = [[e * cp + c for e in range(dp)] for c in range(cp)]
16    cp_groups = [[e * cp + c for c in range(cp)] for e in range(dp)]
17    ep_groups = [list(range(start, start + ep)) for start in range(0, world, ep)]

Thus DP, CP and EP describe different groupings of the same workers. They are not three process pools to add together. The following map assumes one node, zero device offset and no extra Engram replicas. Production choices in the Q&A are a separate configuration, discussed in chapter 11.

B.1 Processes, devices and communication groups

The example launcher examples/_launch.py:26, run_dp, starts one process per engine rank assigned to the node (or uses the current process for one engine). Its RANK/WORLD_SIZE select the node’s share of those ranks. Within each engine, deepinfer/engine/llm_engine.py:368, _init_workers, spawns CP ranks 1 through $C-1$. CP rank zero is a Worker in the engine process itself. The engine’s Python controller is therefore not an additional device worker [94].

WorkerBase in deepinfer/worker/base.py:79 assigns

where $e$ is engine rank, $C$ is CP size and $c$ is CP rank. The example factory’s setup_cuda_visible_devices assigns each engine its contiguous slice of devices; within that slice a worker uses its local index. Figure B.1 shows the two example layouts with four local devices and no offset.

ParallelProcessGroup.create_process_groups builds DP groups across engines at a fixed CP position, CP groups within an engine, and contiguous EP groups:

91# DP group
92dp_ranks = rearrange(ranks, "(replica dp cp) -> (replica cp) dp", dp=dp_size, cp=cp_size)
93self.dp = _split_group(dp_ranks.tolist(), backend=backend, pg_options=options)
94self.cpu_dp_group = _split_group(dp_ranks.tolist(), backend="gloo")
95
96# CP group
97cp_ranks = rearrange(ranks, "(dp cp) -> dp cp", cp=cp_size)
98self.cp = _split_group(cp_ranks.tolist(), backend=backend, pg_options=options)
99self.cpu_cp_group = _split_group(cp_ranks.tolist(), backend="gloo")
100
101# EP groups: contiguous groups of ep_size ranks.
102if ep_size == ranks.numel():
103    self.ep = self.world
104else:
105    ep_ranks = rearrange(ranks, "(edp ep) -> edp ep", ep=ep_size)
106    self.ep = _split_group(ep_ranks.tolist(), backend=backend, pg_options=options)

For DP2/CP2, this produces DP groups $[0,2]$ and $[1,3]$, CP groups $[0,1]$ and $[2,3]$, and the EP group $[0,1,2,3]$. The membership formula is distinct from a workload split: CP workers cooperate on the same engine’s requests, whereas DP engines admit different requests. EP determines where experts run.

Within a CP group, the query rows of each sequence are divided by a mirrored, or zigzag, rule. The reference expansion in model_executor/layers/tilelang_ops/prepare_cp_mirror.py, lines 135–146, cuts a sequence of $q$ rows into $2C$ pieces of $\lceil q∕2C\rceil$ rows; rank $r$ takes pieces $r$ and $2C-1-r$. With $C=2$ and eight rows, rank 0 holds rows 0–1 and 6–7, while rank 1 holds rows 2–5. This is why the scheduler rounds prefill workloads to cp_align$=2C$ (csrc/engine/scheduler.hpp:90; chapter 5). Under causal attention, later rows read more context, so pairing an early piece with a late one plausibly evens out the work. That is our reading of the README’s “balanced CP”, not a measured balance [94].

The general configuration is broader. deepinfer/config.py:150, ParallelConfig, allows Engram parallelism to enlarge the world; the world size is the maximum of that size and DP times CP. The lab deliberately excludes those replicas. CPU control groups and shared-memory queues also exist. Their transport implementation and all distributed failure paths are not covered here [94].

B.2 Follow one step into computation

Figure B.2 is a call-boundary map, not an elapsed-time chart. The scheduler returns work to the engine. It does not call the model runner. The worker separately launches main-model execution and sampling.

The following anchors locate each hop. Paths in this table are relative to deepinfer/, except those starting with csrc/.

The model runner first prepares input metadata. The eager path calls the model; the graph path replays captured work:

326if run_cuda_graph:
327    model_output = self.cuda_graph_runner.replay(cuda_graph_used)
328else:
329    model_output = self.model(kv_caches=kv_caches, input_metadata=input_metadata)

The registered architecture name is DeepseekV41ForCausalLM. It selects the DeepseekVLForCausalLM wrapper, whose forward prepares prompt embeddings when needed. It then delegates to the base wrapper’s DeepseekForCausalLM.forward, which calls the core model [95].

B.3 Follow results back to their owner

The reverse path is not simply Sampler returning a Python answer. Sampling completion permits post-processing; C++ updates sequence state and builds pooled output records; the Python entrypoint clones records it retains (Figure B.3).

The engine installs its callbacks when constructing the driver:

398driver_kwargs = {
399    "engine_callbacks": EngineCallbacks(post_process_fn=self.post_process, pre_sampler_sync_fn=self.pre_sampler_sync),
400}

Start at worker/worker.py:248, _post_process_step. It waits for the sampling event, runs the model runner’s post-functions and invokes the engine callback on the driver. engine/llm_engine.py:984, post_process, enters _process_model_outputs at line 687. That method calls scheduler.process_outputs at line 727, then build_request_outputs at line 743. The binding is in csrc/engine/pybind_output.cpp:41; the builder and pool are in csrc/engine/outputs.hpp:94 and 45. The returned array holds integer addresses. entrypoints/llm.py:181 clones each retained record [5, 96].

These names identify a dependency path, not a guarantee that one step() returns the tokens it just launched. The worker may process step $N$ while step $N+1$ runs. To investigate missing final tokens, follow flush_inflight_reqs; to investigate stale addresses, follow the output pool’s release, not Python reference counts.

B.4 Start from a README feature

The pinned README lists eight features. These tables connect each name to its controls and a source entry point. Batch invariance and deterministic sampling have separate rows because their conditions differ. Settings are not a complete launch recipe; follow the indicated chapter for defaults and limits [1, 97].

Paths are relative to deepinfer/, except csrc/, vendor/ and requirements.txt. The example entry points’ ep and cp arguments choose group sizes; seq_seed and so_args are SamplingParams fields. Names beginning with DEEPINFER_ are environment variables.

B.5 Choose a directory before choosing a file

Coverage labels below describe the book, not the maturity of the software. Deep means a mechanism is traced in detail; partial means selected paths are analysed; locate gives responsibility and an entry point only; outside marks implementation details not analysed. A deep directory entry still does not mean every file in it was reviewed.

For the model modules not expanded in appendix A, start with v4_modules/hc.py: HCLayerMixin mixes the multiple residual streams around attention and feed-forward sublayers, while HCHead folds them back before the output head. hc_blocks.py, hc_lazy.py and mhc_* provide related mixing and deferred work. In v4_modules/engram.py, EngramRuntime fetches sharded n-gram embeddings; related metadata and parameter files describe their representation. These are navigation descriptions, not derivations or evaluations of either model mechanism [95].

Platform dispatch has a useful search boundary: model_executor/layers/tilelang_ops/ops.py imports the backend selected by PLATFORM.tilelang_ops_backend and exports its *_tl factories. An interface module can therefore look platform-neutral while selecting CUDA or Ascend implementations. This is a route into those files, not evidence that their numerics or performance are identical. The independent DeepSeek libraries in vendor/ include DeepGEMM, DeepEP, DeepSelect and FlashMLA; use their pinned submodule revisions when crossing that repository boundary. TileKernels is another independent DeepSeek library, installed through tile-kernels==2.1.0 in requirements.txt, not a vendor/ submodule [97].

B.6 Find the corresponding code in another edition

Begin with a symbol and responsibility, then use line anchors as a convenience. For example, from a DeepInfer checkout:

rg -n 'def execute_model_step|def _post_process_step' deepinfer/worker
rg -n 'class Compressor|class MQA' deepinfer/model_executor/models
rg -n '"schedule"|"process_outputs"' csrc/engine/pybind_scheduler.cpp
rg --files deepinfer/model_executor/parallel_utils

Follow callers as well as definitions: a helper’s existence does not establish that the released engine drives it, as chapter 11’s state-transfer example showed. Recheck defaults and the chosen model/platform branch. To compare a checkout with this book’s evidence, run make check from the book with DEEPINFER_SOURCE=/path/to/DeepInfer. A mismatch detects a changed source edition; it does not identify which behavioural claims remain valid.

As a reading exercise, choose one object: an input position, a compressed row, a grammar matcher or a pooled output address. Locate its creator, its next reader and its release or reuse condition. The directory map supplies entry points; the ownership and evidence methods from the chapters supply the check.