The Iterator Protocol — __iter__ and __next__

The two methods that define an iterator

In Python, anything you can put on the right side of for x in ... is an iterable. The mechanism is simple: an iterable has an __iter__ method that returns an iterator, and an iterator has a __next__ method that returns the next value (or raises StopIteration when there are no more). Lists, tuples, dicts, files, ranges, generator expressions — all support this protocol. Once you understand the two methods, the entire iteration machinery in Python becomes transparent.

Iterable vs. Iterator — the distinction

An iterable can be iterated. An iterator is the thing actually doing the iterating. my_list is iterable; iter(my_list) returns an iterator over it. A for loop calls iter() on its target to get the iterator, then calls next() on the iterator until StopIteration. Most iterators are also iterable — they return themselves from their own __iter__ — but iterables aren't always iterators.

The for-loop, demystified

for x in seq: is, under the hood, almost exactly: it = iter(seq), then a while True: with x = next(it) and a try/except StopIteration: break. Knowing this, you can step through iteration manually when debugging, build your own iterators that compose with for seamlessly, and reason about why some bugs (like iterator exhaustion) happen.

Warning: An iterator can be iterated only once. After StopIteration is raised, calling next() on the same iterator raises StopIteration again forever. If you wrap the same iterator in two for loops, the second loop runs zero iterations. This is the most common source of "why is my second pass empty?" bugs.

Building a custom iterator

Implement __iter__ (returning self, by convention) and __next__ (returning the next value or raising StopIteration). Now for x in my_object works. We'll see in lesson 2 that generator functions provide a vastly easier way to do this — but understanding the protocol first means you can read the source of any standard library iterator and know what's happening.

Code

iter() + next() — the manual way·python

items = [10, 20, 30]
it = iter(items)             # iterator over items

print(next(it))              # 10
print(next(it))              # 20
print(next(it))              # 30

try:
    next(it)
except StopIteration:
    print("exhausted")

# next() also takes a default value to avoid the exception
it = iter([1, 2])
print(next(it, None))        # 1
print(next(it, None))        # 2
print(next(it, None))        # None  — no exception

What a for loop really does·python

items = [10, 20, 30]

# This
for x in items:
    print(x)

# Is equivalent to
it = iter(items)
while True:
    try:
        x = next(it)
    except StopIteration:
        break
    print(x)

Iterator exhaustion — once and you're done·python

items = [1, 2, 3]
it = iter(items)

list(it)                     # [1, 2, 3]
list(it)                     # []   <- already exhausted

# But the underlying ITERABLE can be iterated again — get a fresh iterator
for x in items:
    pass
for x in items:              # works fine — fresh iter() each loop
    print(x)

Custom iterator — implementing the protocol·python

class CountDown:
    def __init__(self, start):
        self.n = start

    def __iter__(self):
        return self            # iterator returns itself

    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        v = self.n
        self.n -= 1
        return v

for x in CountDown(3):
    print(x)
# 3
# 2
# 1

# Works with anything that consumes iterables
print(list(CountDown(5)))    # [5, 4, 3, 2, 1]
print(sum(CountDown(10)))    # 55

Iterable vs Iterator — the distinction in code·python

lst = [1, 2, 3]              # iterable, NOT an iterator
it = iter(lst)               # iterator (calls list.__iter__)

# Iterators support next()
print(next(it))              # 1

# Iterables don't (without iter() first)
try:
    next(lst)
except TypeError as e:
    print(e)                 # 'list' object is not an iterator

# Most iterators are ALSO iterable — they return self from __iter__
print(iter(it) is it)        # True

Exercise

Implement a class Repeater(value, times) that's an iterator yielding value exactly times times before raising StopIteration. (a) Test with for x in Repeater("hi", 3):. (b) Show that list(Repeater(...)) works. (c) Show that calling list() on the SAME Repeater instance twice gives the values once and an empty list the second time (iterator exhaustion).

The Iterator Protocol — iter and next

The two methods that define an iterator

Iterable vs. Iterator — the distinction

The for-loop, demystified

Building a custom iterator

Code

External links

Exercise

Progress

Comments 0