CHAPTER 01

A generation loop

Suppose a function takes a list of integers and returns a score for each possible next integer. You can use it to extend the list: call the function, choose an integer, append it and repeat. This is enough to describe the outer loop of text generation.

Running that loop for many people raises further questions. Which work can be reused? Which requests should run together? When can their memory be released? Four small programs give us the tools to follow a real request through DeepInfer. We will trace admission, scheduling, execution and result ownership, then explain why an empty scheduling queue does not end all of the engine’s work.

Figure 1.1 shows the smallest version: each prediction depends on the token just selected, until an end marker or output limit stops the loop.

1.1 One decision at a time

Language models work with tokens: entries in a vocabulary, each identified by an integer. A token may represent a word, part of a word, punctuation or a special marker. The program that maps text to these integers is a tokenizer. The exact mapping depends on the model.

Our first experiment invents a vocabulary with five entries. Three stand for words; two mark the start and end. The integers and scores are chosen for this exercise.

Run this command from the book directory:

python3 labs/01_generate.py

Before reading the output, follow the first row of SCORES. The starting token has ID zero. The largest score in row zero belongs to token one, Small. That token selects the next row.

8VOCAB = ["<start>", "Small", "steps", "help", "<end>"]
9SCORES = [
10    [0, 9, 1, 0, 0],  # <start> -> Small
11    [0, 0, 8, 1, 0],  # Small   -> steps
12    [0, 0, 0, 7, 1],  # steps   -> help
13    [0, 0, 0, 0, 9],  # help    -> <end>
14    [0, 0, 0, 0, 9],  # <end>   -> <end>
15]
16END = VOCAB.index("<end>")

The loop uses the last token to select a row, chooses the largest score and adds the chosen token to the history. The complete script calls this function and checks its result.

19def generate(max_new_tokens=8):
20    history = [0]
21    output = []
22    for step in range(1, max_new_tokens + 1):
23        scores = SCORES[history[-1]]
24        token = scores.index(max(scores))
25        print(f"step {step}: {VOCAB[history[-1]]} -> {VOCAB[token]}")
26        history.append(token)
27        if token == END:
28            print("stop: end token")
29            break
30        output.append(VOCAB[token])
31    else:
32        print("stop: output limit")
33    print("text:", " ".join(output))
34    return output

Here is the output from running the script:

step 1: <start> -> Small
step 2: Small -> steps
step 3: steps -> help
step 4: help -> <end>
stop: end token
text: Small steps help

The table plays the role of a very small prediction function. Unlike a Transformer, it uses only the last token and has no learned weights. This restriction makes every decision visible.

Choosing the largest score is called greedy selection. A language model usually returns scores called logits. A sampler can turn them into probabilities and draw a token, or use them to make a greedy choice. We will study sampling later. Here, max makes the choice deterministic.

The expression scores.index(max(scores)) finds the first position with the largest score. If two tokens tie, it therefore chooses the earlier one. The else attached to the for loop runs only when the loop reaches its limit without executing break.

The loop needs two ways to stop. The end marker lets the prediction function finish an answer. The output limit stops a function that never chooses that marker. Predict the result of generate(max_new_tokens=2), then try it in a Python session. Does the loop reach the end marker?

1.1.1 What depends on what

Let $x_1,\dots ,x_n$ denote the input tokens. A model with fixed weights $𝜃$ supplies scores:

The selection rule turns $z$ into $x_{n+1}$. The next call then uses the extended sequence. The symbol $f_{𝜃}$ hides the neural network for now; the dependence between calls is what matters.

In ordinary generation, the next call needs the token chosen by the previous call. Knowing that an answer will contain ten tokens does not tell us which ten tokens to feed into ten independent calls. Later, speculative decoding will give us a way to propose several tokens and verify them. It must still respect the model’s generation rule.

1.2 Keep work that will be needed again

A simple implementation could pass the entire growing history to the model on every call. For a causal model, earlier positions cannot depend on tokens that have not yet been generated. Some work on those positions can therefore be saved.

To see the principle without introducing attention, use a prefix sum. Our second prediction function returns the sum of all input integers, modulo ten. The modulo operation keeps the output in the range zero to nine.

Starting with $[1,2,3,5]$, the sum is eleven, so the next integer is one. Appending one makes the sum twelve; the next integer is two. Appending two makes the sum fourteen; the next integer is four.

There are two ways to do this. One sums the whole history each time. The other keeps a running total and adds only the newly generated integer before the next prediction.

python3 labs/02_reuse.py

The first function starts its sum at zero for every prediction:

9def recompute(prompt, steps):
10    history = list(prompt)
11    output = []
12    additions = 0
13    for _ in range(steps):
14        total = 0
15        for token in history:
16            total += token
17            additions += 1
18        token = total % 10
19        history.append(token)
20        output.append(token)
21    return output, additions

The second sums the prompt once. Before each later prediction, it adds the previous output to the saved total:

24def reuse(prompt, steps):
25    total = 0
26    additions = 0
27    processed = 0
28    for token in prompt:
29        total += token
30        additions += 1
31        processed += 1
32    output = []
33    for step in range(steps):
34        if step > 0:
35            total += output[-1]
36            additions += 1
37            processed += 1
38        output.append(total % 10)
39    return output, additions, processed

The full script calls both functions with the same prompt and checks their outputs and addition counts:

prompt: [1, 2, 3, 5]
recompute: [1, 2, 4] | additions: 15
reuse:     [1, 2, 4] | additions: 6
same output: True
history entries: 7 | entries covered by final cache: 6

Both versions produce $[1,2,4]$. The first performs fifteen additions: four before the first output, five before the second and six before the third. The second performs six: four to process the prompt, then one for each of the next two predictions.

For a prompt of length $n$ and $m\ge 1$ output integers, the first version performs

additions. The second performs $n+m-1$.

The final output has been emitted but has not been fed back into the prediction function. The history has seven entries; the running total covers six. If we wanted another output, we would first add the last emitted integer. Figure 1.2 separates the two boundaries. DeepInfer adds another: when a device result becomes available to the host. Its asynchronous scheduler must track these boundaries separately.

1.2.1 From a running total to a cache

The running total is a cache: saved information used to avoid repeating work. It is useful because it contains enough information for our next update. Appending a token preserves that property. Editing an earlier token without repairing the total would break it.

Transformer attention needs a richer cache. Earlier positions provide arrays called keys and values, which later positions use to compute their representations. Saving these arrays gives a key–value cache, usually shortened to KV cache. We will derive the arrays and their memory cost in later chapters. The running sum occupies constant space. A full-attention KV cache grows with the number of processed tokens, and each decode step still reads keys and values across that context. Caching avoids rebuilding them; it does not make attention a constant-cost operation. A sliding window bounds the context read by that attention layer, which changes both its memory use and what it can see.

Processing the supplied prompt is called prefill. Extending the sequence using saved state is called decode. In a conventional cached generation loop, the final prompt position supplies the scores for the first output token. The next model call processes that token to predict the second.

These names identify work, not a guarantee about how many device calls it takes. A long prompt may be split across several steps. A speculative step may verify several candidate tokens. Our simple loop gives us a starting point for understanding both changes.

1.3 Rebuild a lost window

How much history do we need to recover a cache? Change the predictor to the sum of the last three processed integers, modulo ten. A bounded deque holds that window: appending a fourth integer discards the oldest. This gives us the behaviour of a three-slot ring buffer without writing its index arithmetic.

python3 labs/04_window.py

Predict what happens if we lose the window after four outputs. The token history is still available, together with the number of entries already processed:

11def generate(prompt, steps, width=3):
12    history = list(prompt)
13    window = deque(prompt, maxlen=width)
14    processed = len(prompt)
15    output = []
16    for step in range(steps):
17        if step > 0:
18            window.append(output[-1])
19            processed += 1
20        token = sum(window) % 10
21        assert list(window) == history[max(0, processed - width):processed]
22        assert token == sum(history[max(0, processed - width):processed]) % 10
23        history.append(token)
24        output.append(token)
25    return history, output, window, processed
26
27
28def rebuild(history, processed, width=3):
29    return deque(history[max(0, processed - width):processed], maxlen=width)
30

The script clears the window, calls rebuild and checks it against the saved state. It also shows what happens if we take the wrong end of the history:

generated: [0, 8, 3, 1]
history entries: 8 | processed entries: 7
window before loss: [0, 8, 3]
rebuilt from processed suffix: [0, 8, 3]
same window: True
wrong suffix (includes latest output): [8, 3, 1]
next token after processing latest output: 2

The four outputs are $[0,8,3,1]$. At the checkpoint, seven entries have been processed, and the window is $[0,8,3]$. The latest output, 1, has been emitted but not processed. Taking the last three entries of the entire history would include it too soon. To continue after recovery, first process that 1: the window becomes $[8,3,1]$, whose sum predicts 2.

Recovery is exact here because the state contains raw integers. The last three processed entries determine the whole window, regardless of the earlier history. DeepInfer’s sliding-window attention (SWA) bounded replay tackles a related problem, but reconstructs an approximate attention cache: hidden states inside the replayed suffix can depend on earlier context. Chapter 9 will explain that approximation alongside cross-layer sharing. Assessing it requires model-quality checks as well as checks of buffer contents [2].

1.4 Give an empty slot to another request

So far, one request has owned the loop. Now suppose three requests are waiting. For this experiment, A needs two output tokens, B needs four and C needs one. The engine can advance at most two requests in a round.

We make strong assumptions to isolate one scheduling decision. All requests arrive at the start. Each active request gains exactly one token per round. Every round costs one abstract time unit, however many slots are occupied. There is no prefill cost or memory limit.

A fixed-batch policy admits A and B together, waits until both finish, then admits C. A second policy refills an empty slot at the start of each round. Predict when C will finish under each policy.

python3 labs/03_batch.py

The waiting queue supplies requests in arrival order. An active slot holds a request name and its remaining work; None means the slot is empty. The refill switch controls admission at the start of each round.

9from collections import deque
10
11JOBS = [("A", 2), ("B", 4), ("C", 1)]
12
13
14def simulate(refill):
15    waiting = deque(JOBS)
16    active = [None, None]
17    finished = {}
18    rounds = 0
19    unused = 0
20    while waiting or any(job is not None for job in active):
21        if refill or all(job is None for job in active):
22            for slot, job in enumerate(active):
23                if job is None and waiting:
24                    name, length = waiting.popleft()
25                    active[slot] = [name, length]
26        rounds += 1
27        names = [job[0] if job else "-" for job in active]
28        print(f"round {rounds}: {' '.join(names)}")
29        for slot, job in enumerate(active):
30            if job is None:
31                unused += 1
32                continue
33            job[1] -= 1
34            if job[1] == 0:
35                finished[job[0]] = rounds
36                active[slot] = None
37    print(f"rounds: {rounds} | unused slot-rounds: {unused}")
38    print("finish:", ", ".join(f"{name}={finished[name]}" for name, _ in JOBS))
39    return rounds, unused, finished

The script calls simulate once with refill=False and once with refill=True, and checks both results:

Fixed batches
round 1: A B
round 2: A B
round 3: - B
round 4: - B
round 5: C -
rounds: 5 | unused slot-rounds: 3
finish: A=2, B=4, C=5

Refill after each round
round 1: A B
round 2: A B
round 3: C B
round 4: - B
rounds: 4 | unused slot-rounds: 1
finish: A=2, B=4, C=3

A finishes after round two under both policies. With fixed batches, its slot stays empty while B runs in rounds three and four. C must wait until round five. Refilling the empty slot lets C run in round three alongside B. Figure 1.3 compares the two schedules.

Both policies do seven units of useful token work. Fixed batches provide ten slot-rounds across five rounds and leave three unused. Refilling provides eight across four rounds and leaves one unused. B still finishes after round four; the shorter request benefits from the earlier admission.

Allowing batch membership to change between generation steps is often called continuous batching. Admitting C in an engine also requires prompt processing and enough memory for its state.

Notice that the scheduling policy never reads the remaining length when choosing the next request. The simulator uses that length only to decide when the simulated request finishes. A real engine generally discovers completion as generation proceeds.

1.5 What an engine must remember

The programs have acquired several kinds of state: token history and an output limit, reusable computation with a processed boundary, and a waiting queue with active slots and completion records.

For each request, an engine needs to answer at least four questions:

1. Which input and output tokens belong to it?
2. How much of that sequence has the model processed?
3. Which memory holds the state needed to continue?
4. Is it waiting, running or finished?

The engine also needs shared decisions: which requests to advance and how much work to admit. A scheduler makes those choices. A worker carries out the selected model work. The separation lets us discuss a scheduling policy without rewriting the prediction function.

There is a further complication when computation runs on another device. Submitting work may return before that work finishes. A request that has left a scheduling queue may still have a device operation using its memory. Resource lifetime therefore depends on completion, as well as logical request status. We will examine this in the chapter on overlapping host and device work.

1.6 Follow state through DeepInfer

DeepInfer’s README says it was originally built primarily on vLLM; this book has not identified the fork point. Its asynchronous execution makes the loop more demanding. Its zero-overhead batch scheduler (ZOS) overlaps host scheduling with device computation. The deployment Q&A credits SGLang and NanoFlow for earlier implementations of this approach [1, 2].

Follow one request through the ordinary offline interface: valid text token IDs, with no multimodal processor or structured-output grammar. We will read its admission, execution and result path, then revisit what happens when execution overlaps. This is a walkthrough of the inspected code, not a recorded engine run. Figure 1.4 gives the route. The arrows cross responsibility boundaries: the entry point prepares requests, the engine coordinates them, the scheduler selects work and the worker executes it.

1.6.1 A request becomes persistent engine state

LLM.generate submits each input through _add_request before entering its generation loop. The helper begins by separating the request’s sampling settings from the caller’s object, assigning an identifier and making a token array:

151sampling_params = copy.deepcopy(sampling_params)
152request_id = str(next(self.request_counter))
153if prompt_token_ids is None:
154    if prompt is None:
155        raise ValueError("prompt and prompt_token_ids cannot both be None")
156    prompt_token_ids = self.llm_engine.tokenizer.encode(prompt)
157prompt_token_ids = np.asarray(prompt_token_ids, dtype=np.int32)

For our token-ID input, the tokenizer branch is skipped. deepcopy creates independent sampling settings even if the caller supplied the same object for several requests; converting to int32 establishes the array’s element type. It does not promise another copy when the input already has that type. These are different guarantees. The helper passes the identifier, settings and token array to LLMEngine.add_request.

After validation and input preparation, the engine creates a C++-backed Sequence. The sequence_args dictionary carries the identifier, settings and other request metadata prepared earlier in that method:

594seq = Sequence(
595    prompt_token_ids=prompt_token_ids,
596    compressed_prompt_token_ids=compressed_prompt_token_ids,
597    **sequence_args,
598)

The two token arrays serve different input representations. Their model-specific contents are not needed to follow admission: both belong to the request whose state will survive successive scheduling rounds. For our request, no grammar construction is pending, so the next fragment takes its final else:

620def _enqueue(s: Sequence) -> None:
621    if s.so_matcher is not None and s.so_matcher.is_aborted():
622        s.status = SequenceStatus.FINISHED_SO_FAILED
623    self.scheduler.add_sequence(s)
624    if self.multi_modal_processor is not None:
625        self.multi_modal_processor.add(request_id, multi_modal_data)
626
627if so_pending:
628    self.so_matcher_builder.submit(seq, sampling_params.so_args, _enqueue)
629else:
630    _enqueue(seq)

Here so_pending means that a structured-output grammar still needs to be built. Our case also has no multimodal processor, so neither extra path runs. The call to add_sequence crosses into C++:

135void add_sequence(std::shared_ptr<Sequence> seq) {
136    seq->request.arrival_time = get_timestamp_ms();
137    if (engine_type == EngineType::NORMAL) {
138        seq->reserve_output_tokens();
139        seq->cache.block_table.reserve(_block_table_capacity(*seq));
140    }
141    waiting.push_back(std::move(seq));
142}

The scheduler records an admission timestamp and appends the sequence to waiting. The C++ shared_ptr is an ownership handle; std::move transfers that handle into the queue without copying the sequence’s token history. The arrow in seq->request accesses a member through that handle. For a normal offline engine, the method also reserves capacity in vectors for output tokens and block references. Reserving a vector’s capacity is not allocating the device’s KV cache. That resource decision belongs to later scheduling and cache management.

This boundary explains why generate can submit many requests before running any model work. Admission records available work; scheduling decides what fits now. A useful consequence of this separation is that a new admission policy need not alter the model’s prediction function. We infer that benefit from the division of responsibilities, rather than from a stated design motive.

1.6.2 Generated length does not measure completed work

80void append_token_id(int token_id) { token_ids.push_back(token_id); }
81int get_len() const { return static_cast<int>(token_ids.size()); }
82int get_prompt_len() const { return prompt_len; }
83int get_output_len() const { return get_len() - prompt_len; }
84const std::vector<int> &get_token_ids() const { return token_ids; }
85int get_last_token_id() const { return token_ids.back(); }
86
87std::vector<int> token_ids;
88int prompt_len{0};

The persistent Sequence contains input tokens, generation settings, cache bookkeeping and scheduling state. The scheduler holds this request object across steps. These methods belong to its SequenceData token history. Output length is total history length minus prompt length. It counts generated tokens, not cached positions or device work that has completed. Our running-total experiment shows why that distinction matters [6].

The nearby SequenceStatus enum includes WAITING, RUNNING, SWAPPED and several reasons for finishing. An enum value does not tell us which transitions the scheduler actually takes. For this request, follow admission into RUNNING and eventual completion. Chapter 5 examines what changes when memory pressure interrupts that route [8].

1.6.3 A scheduling decision becomes model work

Now return to the engine’s step. Its call to the C++ scheduler returns per-request metadata and one object describing decisions for the whole batch:

885seq_metadata_list, scheduler_outputs = self.scheduler.schedule(
886    self.sequence_metadata_pool
887)  # Will also free the finished sequence groups.

Here, metadata means lengths, positions and references needed to interpret arrays. SequenceMetadata describes selected work for a request in this step; SchedulerOutputs carries batch decisions, including whether the work is prompt processing and which sequences were selected. Neither is the user’s completed answer. A correct token array with incorrect positions can still produce an incorrect calculation.

After preparing state transfers, the engine calls _execute_model_async_step. Its implementation shows exactly what crosses the worker boundary:

961def _execute_model_async_step(
962    self, seq_metadata_list: list[SequenceMetadata], scheduler_outputs: SchedulerOutputs
963) -> np.ndarray[np.uint64]:
964    with cpu_timer.record("run_workers"):
965        request_outputs = self._run_workers(
966            "execute_model_step",
967            driver_kwargs=dict(
968                seq_metadata_list=seq_metadata_list,
969                scheduler_outputs=scheduler_outputs,
970            ),
971            is_prompt=scheduler_outputs.prompt_run,
972            is_placeholder=scheduler_outputs.is_empty(),
973        )
974    self.deferred_release.collect_request_outputs(request_outputs)
975    return request_outputs

_run_workers delegates to the driver worker’s dispatch method. The per-request metadata and scheduler result travel in driver_kwargs; the prompt and placeholder flags are separate arguments. A placeholder means no real sequences were selected, but the worker still follows an execution path. An empty current batch therefore does not imply that all earlier work is done.

The returned array is registered with deferred_release before being returned to the caller. This connects the execution path to an ownership rule: these outputs are temporary, and the engine is already tracking their eventual release. We will inspect the copy that makes a result safe to keep below [5].

The comment in Listing 1.12 still says “sequence groups”. vLLM v0.6.3 defines SequenceGroup in vllm/sequence.py and describes freeing finished sequence groups in its engine’s step docstring. In the inspected DeepInfer engine, the scheduler instead handles Sequence objects; its Python and C++ sources define no SequenceGroup type [11, 12, 8]. This is consistent with inherited terminology, though it does not establish the exact fork point. Read the types and control flow before treating an old comment as a current design description.

1.6.4 One call can handle work from two steps

The opening of Worker.execute_model_step’s docstring gives the order:

96"""The unified entry for model execution: every step (prompt/decode, ZOS on/off,
97placeholder) goes through this Forward-Post_process-Sample round.
98  1. Forward current step (main model)
99  2. Post-process previous step
100  3. Sample current step
101  4. Post-process current step: in place for prompt / non-ZOS steps;
102     ZOS decode steps defer it to the next step to overlap with forward

The implementation makes the delayed state explicit. After current model work, it first processes the state saved from the previous call:

143# 2. post process prev step
144request_outputs = self._post_process_step(self.prev_step_state)

It then samples the current work and packages its metadata, sample output and completion event in curr_step_state. The final branch decides whether that package is processed now or retained:

183if is_prompt or not envs.DEEPINFER_ZERO_OVERHEAD_SCHEDULING:
184    request_outputs = np.concatenate([request_outputs, self._post_process_step(curr_step_state)])
185    self.prev_step_state = None
186else:
187    self.prev_step_state = curr_step_state
188
189return request_outputs

For prompt work or disabled ZOS, the returned outputs include current-step post-processing. For a ZOS decode step, prev_step_state instead becomes the current package; the returned outputs came from the earlier package. The method’s return is therefore not a universal “current prediction is ready” boundary. Chapter 6 will open the completion-event and metadata-correction mechanisms behind this branch.

1.6.5 A queue can be empty while results are still pending

LLM.generate adds requests through _add_request and enters _run_engine. Here is the loop that collects their results:

175outputs: dict[str, list[RequestOutput]] = defaultdict(list)
176while True:
177    unfinished = self.llm_engine.has_unfinished_requests()
178    step_outputs = self.llm_engine.step() if unfinished else self.llm_engine.flush_inflight_reqs()
179    for raw_output in step_outputs:
180        # clone step_outputs since step_outputs will be released in the next step
181        output = RequestOutput.clone(raw_output)
182        outputs[output.request_id].append(output)
183        if output.finished and use_tqdm:
184            pbar.update(1)
185    if not unfinished:
186        break
187request_ids = sorted(outputs, key=int)

The conditional call is the important part. Once the engine reports no unfinished requests, the loop still calls flush_inflight_reqs, collects its outputs and only then breaks. The docstring of LLMEngine.flush_inflight_reqs, line 978, says: “Flush and post-process all requests in flight under ZOS” (author’s translation). Logical completion and completion of result handling are separate events [5].

The engine’s flush delegates to the worker’s flush method. Its body consumes exactly the state retained by the branch above:

281request_outputs = self._post_process_step(self.prev_step_state)
282self.prev_step_state = None
283return request_outputs

_post_process_step returns an empty address array when its argument is None. Otherwise it waits for sampling completion, prepares the model results and calls the engine’s post-processing callback. The engine updates sequence state through scheduler.process_outputs, then builds pooled request outputs. The outer loop can thus collect the same form of result from an ordinary step or the final drain. Clearing prev_step_state prevents that retained package from being consumed twice [5, 7].

1.6.6 An output has an owner and a lifetime

The comment immediately before RequestOutput.clone(raw_output) belongs to the call site above. It explains why the outer loop copies each output. The stronger ownership contract appears in LLMEngine.step:

861"""Run one inference step and return the generated outputs.
862
863Returns:
864    np.ndarray[np.uint64]: entrypoint -> a list of RequestOutput objects.
865
866Note:
867    The returned RequestOutput objects are managed by RequestOutputPool.
868    RequestOutput will be automatically released in the next step() call.
869    If you need to retain them beyond the current step, you must call pool.clone(list[RequestOutput]) to obtain copies,
870    and release them via pool.release(list[RequestOutput]).
871"""

Notice the return type: np.ndarray[np.uint64]. Its entries are integer addresses of C++ RequestOutput objects, not Python references to those objects. Keeping the array, or putting its integers in a Python list, preserves the addresses but does not keep the pointed-to objects alive. The C++ pool can destroy and reuse them at the next step [9].

The loop therefore calls RequestOutput.clone(raw_output) while each address is still valid. This static method is declared in deepinfer/_C/engine.pyi, line 182. Its C++ binding copies the object at the supplied address into a new object owned by the Python wrapper [10]. The docstring’s reference to pool.clone does not match that exposed API: use the entry-point call and the binding to identify the operation. A Python reference protects the copied object; an integer address does not protect the pooled original.

Cloning during collection preserves the result across the next pool release. Chapter 4 returns to the binding and pool implementation in Section 4.3.9, alongside the ownership rules for cached state. Here the caller needs one rule: copy a result while its address is valid.

1.6.7 Trace the same request across the boundaries

Our ordinary request now has a complete route. After admission it is a Sequence in waiting. When selected, its per-step metadata travels through the engine to the worker. Under ZOS decode, the worker retains the current step’s package and reports completed earlier work. Post-processing updates the request and creates output addresses; the offline collector copies them before the pool may reuse their storage.

For a concrete branch trace, suppose this request remains active after its last prompt chunk and through two ZOS decode calls. Name the decode packages $D_1$ and $D_2$. With no pending state on entry to that prompt call, the printed worker code gives:

The $D_1$ call can return no output despite doing model work. The $D_2$ call can return $D_1$’s results while retaining its own. A later flush consumes whichever package remains. This predicts branch behaviour, not particular token values or the number of calls until the request finishes.

Three different objects therefore answer three different questions:

To investigate a missing final result, first compare the worker’s pending state with the outer loop’s flush path. To investigate results that change after a later step, inspect collection and cloning before changing sampling. These are different failure hypotheses because the execution and ownership boundaries are different. Chapters 4–6 will open the cache, scheduling and overlap boxes in the same map.

1.7 Experiments to try next

Keep the original scripts and saved outputs as a reference. Make each change in a scratch copy, and update its assertions when you deliberately change the expected result.

1. Change a score so that the first program repeats steps. Does it reach the end marker, or stop at the output limit?
2. Change the prefix-sum prompt and generate five integers. Check that both versions agree. Predict their addition counts before running them.
3. In the window experiment, use width four and a two-integer prompt. Rebuild after the first output and again after several outputs. Does each recovered window predict the same next integer after processing the latest output?
4. Give A and B the same length in the batching experiment. Does refilling still reduce the number of rounds?
5. Give C a later arrival time. Which part of the simulator must change? Is an empty slot always evidence of a bad scheduling decision?
6. Follow the printed worker branch with ZOS disabled, then enabled for decode. Which step’s outputs can the caller receive? Explain what final flush consumes in each case.
7. Suppose the entry point keeps raw output addresses instead of cloning. Locate the operation that invalidates that choice. Why does keeping the addresses in a Python list not fix it?