Iterator Patterns — Pipelines and the Pythonic Stream
~20 min · pipeline, stream, composition
Level 0Curious
0 XP0/93 lessons0/23 achievements
0/100 XP to next level100 XP to go0% complete
Composing iterators into pipelines
Once you can write generators and use itertools, you can build pipelines: each stage is a generator that takes one iterable and yields a transformed iterable. They compose like Unix pipes. The whole pipeline is lazy — nothing happens until the consumer pulls a value, and then exactly one value flows through every stage.
The shape that beats nested loops
You see this in good Python a lot: instead of one big loop that does five things, you write five tiny generator functions that each do one thing, and chain them. parsed = parse(stripped(non_empty(lines(path)))). Each layer is testable, each is reusable, and the data only walks through once.
Avoiding the materialize-everything trap
The mistake is breaking the chain by calling list() in the middle. list(stage_3(stage_2(stage_1(...)))) is fine — that's the consumer. But stage_3(list(stage_2(stage_1(...)))) defeats the purpose: it materializes the intermediate result, eating memory you didn't have to. Keep the laziness end-to-end until you actually need a list.
Pipelines and side effects don't mix well
If a generator's body has side effects (logging, mutating shared state, raising errors), those side effects happen at iteration time, not at definition time. g = my_generator() doesn't run anything yet. The first next(g) runs the body up to the first yield. This is fine if you understand it; it's a debugging nightmare if you don't.
War Story: A common bug — wrapping a generator in asyncio.wait_for on its __anext__ call. When the wait times out, the cancellation finalizes the generator, and any subsequent next raises StopIteration instead of recovering. cwkPippa's adapter doesn't do that, deliberately. The lesson: timeouts on iteration are subtle, treat them like load-bearing structural code, not a casual decorator.
Code
Pipeline — five-stage stream·python
def lines_of(text):
yield from text.splitlines()
def strip_each(lines):
for line in lines:
yield line.strip()
def drop_empty(lines):
for line in lines:
if line:
yield line
def first_word(lines):
for line in lines:
yield line.split(" ", 1)[0]
def long_only(words, min_len=4):
for w in words:
if len(w) >= min_len:
yield w
source = """hello world\n\n foo bar\nhi there\nlongword something"""
result = list(long_only(first_word(drop_empty(strip_each(lines_of(source))))))
print(result) # ['hello', 'longword']
Lazy beats eager — when the source is huge·python
import itertools as it
def naturals():
n = 1
while True:
yield n
n += 1
def squares(nums):
for n in nums:
yield n * n
def under_limit(nums, limit):
for n in nums:
if n >= limit:
return
yield n
# Pipeline: naturals -> squares -> under 10000
result = list(under_limit(squares(naturals()), 10000))
print(result)
# [1, 4, 9, 16, 25, 36, 49, 64, 81, 100, ...]
# stops at 99*99 = 9801 (next would be 10000)
The bug — generators don't run at definition·python
def risky():
print("about to fail")
raise ValueError("boom")
yield 1 # never reached, but the function is still a generator
g = risky() # NO output, NO error — the body hasn't run yet
print("got generator")
try:
next(g)
except ValueError as e:
print("raised at iteration time:", e)
# about to fail
# raised at iteration time: boom
Materializing in the middle defeats the laziness·python
import itertools as it
def big_source():
for i in range(10**6):
yield i
# BAD — materializes the whole million-item list before slicing
result_bad = list(it.islice(list(big_source()), 5))
# GOOD — stays lazy end to end
result_good = list(it.islice(big_source(), 5))
print(result_bad == result_good) # True — same answer
# But result_bad allocated a million-item list along the way
Build a 4-stage generator pipeline over a multi-line string of CSV-like data: 'name,age\nalice,30\nbob,25\ncharlie,99\n,'. Stage 1: yield non-empty stripped lines. Stage 2: yield only lines with at least one comma. Stage 3: yield (name, int(age)) tuples — skipping any line where conversion fails. Stage 4: yield only tuples where age >= 18. Consume into a list. Verify the pipeline works on the input and the failing line is silently skipped.
Progress
Progress is local-only — sign in to sync across devices.