Python / Reliable programs / Lesson 10

Comprehensions and iterator pipelines

Express transformations compactly while keeping evaluation order, one-pass consumption, and memory behavior explicit.

About 4 min readContent reviewed 2026-10-23

On this page

Lesson goals

Read a comprehension as a loop
Validate pipeline assumptions

Read a comprehension as a loop

A list comprehension builds a new list from an iterable, optionally filtering elements. Read [expression for item in source if condition] as visiting each item, checking the condition, and appending the expression for those that pass. It is appropriate for a short transformation without unrelated side effects. A nested comprehension follows its for clauses from left to right, just like nested loops, which can multiply the number of visits. If understanding it requires tracing several branches or handling exceptions inline, an ordinary loop is usually the clearer design. Compact syntax earns its place by making intent more visible.

Iterable and iterator are different roles

An iterable can provide an iterator through iter. An iterator keeps progress and supplies the next value through next, eventually raising StopIteration. A list can create a fresh iterator each time, while an iterator object is ordinarily consumed once. Passing the same iterator to two consumers makes them share progress. Converting a source to a list can provide repeatable traversal, but costs memory and may be impossible for an infinite source. Choose whether a function promises to consume an input once or to require a reusable collection; callers need that information to avoid subtle missing-data behavior.

Transform and filter deliberately

The first example computes squares only for even inputs. Filtering is evaluated before the output expression for each input, so discarded inputs do not contribute results. Dictionary and set comprehensions use similar syntax with different output semantics: duplicate dictionary keys overwrite earlier values, while a set removes duplicate values. These are meaningful choices rather than stylistic variations. If key collisions represent corrupt input, a dictionary comprehension that silently keeps the latest record is the wrong implementation; use an explicit loop that detects and reports duplicates. Always explain what happens when the input is empty or contains repeated values.

python Transform and filter deliberately — worked example

values = [1, 2, 3, 4, 5]
even_squares = [value * value for value in values if value % 2 == 0]
print(even_squares)
print({value: value * value for value in [2, 3]})

Expected output

[4, 16]
{2: 4, 3: 9}

Only two and four pass the filter. The dictionary example associates each explicit input with its square and retains insertion order.

Consume lazily and in bounded pieces

Generator expressions compute values as requested instead of eagerly constructing a complete list. itertools.islice can take a bounded prefix from a source, and functions such as sum can consume a stream incrementally. Laziness reduces retained intermediate data but does not automatically reduce total work or make an operation asynchronous. A lazy computation may also raise its error later, during consumption rather than creation. The second example demonstrates one-pass progress explicitly: the first slice takes two items, and the second consumer receives only what remains. Treat iterator ownership as part of the function contract, especially in a pipeline with multiple stages.

python Consume lazily and in bounded pieces — worked example

from itertools import islice

source = iter([10, 20, 30, 40])
print(list(islice(source, 2)))
print(list(source))
print(list(source))

Expected output

[10, 20]
[30, 40]
[]

The first consumer advances the shared iterator by two items. The next drains it; the final traversal confirms exhaustion rather than restarting the original list.

Validate pipeline assumptions

Small iterator tools compose well when their contracts agree. zip stops at the shortest input by default; zip(..., strict=True), available throughout this course’s supported versions, rejects differing lengths when equal length is required. enumerate supplies a count without repeated index lookup. Sorting consumes all inputs and returns a list, so placing sorted in the middle of a pipeline removes its streaming memory advantage. Review each stage for whether it consumes, buffers, reorders, or duplicates data. Test a single-pass iterator as well as a list; code that accidentally traverses its input twice often appears correct with lists and fails on real streams.

Common errors and fixes

A second pass unexpectedly returns no data

Why it happens: The input is an exhausted iterator rather than a reusable collection.
How to fix it: Consume once or materialize deliberately within a documented size limit.
Prevent it next time: Test functions with iter(list_data), not only list_data.

zip silently drops unmatched values

Why it happens: Default zip stops at the shortest iterable.
How to fix it: Use strict=True when unequal lengths violate the contract.
Prevent it next time: Include mismatched-length cases in validation.

A stream suddenly uses large memory

Why it happens: An intermediate list or sort eagerly retains all values.
How to fix it: Remove unnecessary materialization or specify a bounded batch strategy.
Prevent it next time: Mark buffering stages during pipeline design.

Check your understanding

1. Create a comprehension for stripped nonempty strings from [" a ", "", " b"].

Show worked solution

Use [text.strip() for text in values if text.strip()]. It returns ["a", "b"]. For costly normalization or more complex policy, use an explicit loop that computes stripped once and appends only nonempty results, avoiding duplicated work while keeping the decision clear.

2. What should a function do if it needs two traversals of an input iterator?