Python / Maintainable systems / Lesson 14

Generators, delegation, and bounded streams

Build lazy producers, understand suspension and cleanup, and make backpressure and resource ownership visible in streaming designs.

About 5 min readContent reviewed 2026-10-23

On this page

Lesson goals

A generator is a suspended computation
Cleanup and failure timing

A generator is a suspended computation

A function containing yield produces a generator object when called. Its body advances when the consumer requests the next value, suspends at yield, and preserves local state for resumption. Reaching the end signals exhaustion. This is different from returning a list, which completes all element production before the caller receives it. A generator expresses a sequence of work in natural order without manually implementing a stateful iterator class. It still executes in the consumer’s synchronous flow unless another concurrency mechanism is introduced. Lazy production is a control-flow technique, not a promise of parallel execution or automatic speed.

Consumption controls progress

A pull-based consumer requests one value at a time, so the producer cannot run arbitrarily far ahead through ordinary iteration. This provides a simple form of backpressure for synchronous pipelines. Intermediate stages may still buffer or duplicate data, and a consumer that stores every output removes the memory benefit. A source may be finite, very large, or infinite; a finite request does not imply a finite underlying stream. Use a clear stopping condition, such as islice for a bounded prefix. If producing one value performs substantial work, cancellation only becomes observable at the points the design actually checks or yields.

Produce a bounded sequence

The first example emits running totals and preserves the accumulator between requests. Calling the function creates a generator but does not yet consume the input values. Taking two outputs executes only the work needed for those outputs; the remaining consumer resumes the same computation. This statefulness is useful for streams but must be documented when a generator is handed to more than one consumer. A second independent consumer does not receive a private copy of progress. To support replay, store an appropriate snapshot or create a new generator from a repeatable source, acknowledging the additional memory or repeated-work cost.

python Produce a bounded sequence — worked example

from itertools import islice

def running_totals(values):
    total = 0
    for value in values:
        total += value
        yield total

stream = running_totals([2, 3, 5])
print(list(islice(stream, 2)))
print(list(stream))

Expected output

[2, 5]
[10]

The suspended total is five after two requests. Resuming adds the final five and yields ten; no restart occurs.

Delegate without losing intent

yield from delegates iteration to another iterable and is more than syntactic flattening in the full generator protocol. For a simple pipeline it clearly says that every value of a subsource should be forwarded. The second example forwards two finite groups and preserves their order. It deliberately avoids recursive flattening of arbitrary input, which would need rules for strings, bytes, mappings, cycles, and infinite nesting. A useful generator should have a precise input contract instead of guessing which objects the caller intended to flatten. When delegation includes resource-owning generators, understand how exhaustion, close, and exceptions propagate through the chain.

python Delegate without losing intent — worked example

def concatenate(groups):
    for group in groups:
        yield from group

print(list(concatenate([[1, 2], [], [3]])))

Expected output

[1, 2, 3]

Delegation forwards each group in order, and the empty group contributes no values. This is one-level concatenation under a narrow sequence-of-groups contract.

Cleanup and failure timing

A generator may hold a file open while suspended inside a with block. If a consumer stops early, do not rely on prompt garbage collection for resource cleanup; make ownership explicit with a context manager or an appropriate closing wrapper. A finally block can release resources when the generator is closed. Errors in a lazy stage occur during iteration, so constructing the pipeline successfully does not validate every future record. Test normal exhaustion, an early stop, an empty source, and a failure after some values were already yielded. Consider whether partially emitted results are acceptable or whether the application requires validation before publication.

Common errors and fixes

A generator object produces no visible work yet

Why it happens: Calling the generator function only creates its suspended iterator.
How to fix it: Consume it with iteration or next under a bounded plan.
Prevent it next time: Distinguish pipeline construction from execution in tests.

A resource remains open after an early break

Why it happens: The suspended generator still owns the resource.
How to fix it: Close under explicit ownership or wrap lifetime in a context manager.
Prevent it next time: Include early-termination cleanup in the contract and tests.

An infinite source hangs during list conversion

Why it happens: Materialization requests all future values.
How to fix it: Consume a bounded prefix or a finite domain-specific window.
Prevent it next time: Record finiteness assumptions for every consuming operation.

Check your understanding

1. Write the reasoning for a generator that yields only the first three running totals of [1, 2, 3, 4].

Show worked solution

Use the running_totals generator and consume list(islice(stream, 3)). The results are [1, 3, 6]. The fourth input has not been processed by that consumption; a later request on the same stream yields 10, demonstrating suspended state rather than precomputed output.

2. Why can catching errors around generator creation miss malformed input?