Iterators and Generators

for has appeared in nearly every chapter of the Python for Beginners series, always treated as magic. It is not: loops run on two small methods you can write yourself. Chapter 18 of this 24-chapter series opens up the iteration protocol, then shows generators — the yield keyword that makes an iterator almost effortless — and the lazy pipelines that process huge files without loading them into memory.

The iteration protocol

Anything a for loop accepts is an iterator: an object with __iter__ (returning itself) and __next__ (one value per call, then a polite refusal). Here is a counter built from scratch:

class Counter:
    def __init__(self, start, stop):
        self.current = start
        self.stop = stop

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += 1
        return value
for n in Counter(1, 4):
    print(n)

Expected output:

1
2
3

The last value is never returned: when there is nothing left, __next__ raises StopIteration and the loop ends quietly. Returning a sentinel such as None would be wrong, because None can be a perfectly good value.

Desugaring the for loop

You have just written the two methods a loop needs, so it is worth seeing what the loop itself is. Python’s for is closer to JavaScript’s for...of than to a counted for(i = 0; ...): it asks for values until the object runs out, exactly as in this manual form.

it = iter(Counter(1, 4))
while True:
    try:
        n = next(it)
    except StopIteration:
        break
    print(n)

Expected output:

1
2
3

Every for x in y is that block, written for you. The two built-ins are the moving parts: iter(obj) calls __iter__ and next(it) calls __next__, raising StopIteration when the values run out — next(iter([10, 20, 30])) on its own would reply 10. A class with only __next__ is not enough — a loop then refuses it with TypeError: 'Counter' object is not iterable.

Generators: yield instead of return

Hand-writing those methods is still tedious. A function containing yield is a generator function: calling it produces a generator object, and each yield hands one value to the caller and freezes the function’s state until it is asked again.

def countdown(start):
    while start > 0:
        yield start
        start -= 1
gen = countdown(3)
print(next(gen))
print(list(gen))
print(list(gen))

Expected output:

3
[2, 1]
[]

Three lessons in three lines. Nothing happens when countdown(3) is called — the body runs only on demand (put a print() inside and nothing appears until you request a value). Execution resumes exactly where it paused, local variables intact. And a generator is one-shot: once exhausted it answers [] forever, so call it again when you need the values again.

Pipelines and lazy evaluation

Generators shine when composed: each stage takes an iterable and yields a transformed one, so a reader, a filter and an aggregator chain together without building a single intermediate list.

def first_line(text):
    yield text.splitlines()[0]

def upper(lines):
    for line in lines:
        yield line.upper()

doc = "alpha\nbeta\ngamma"
print(list(upper(first_line(doc))))

Expected output: ['ALPHA']. This is the instinct behind mapping over an array in JavaScript, but nothing is computed early — which is the difference between processing a file and surviving it. Compare a generator with the equivalent materialised list:

import sys

def squares(limit):
    for i in range(limit):
        yield i * i
gen = squares(1_000_000)          # one million values, nothing computed yet
values = list(range(100_000, 200_000))
print("generator object:", sys.getsizeof(gen), "bytes")
print("list plus integers:", sys.getsizeof(values) + sum(sys.getsizeof(v) for v in values), "bytes")

Expected output:

generator object: 200 bytes
list plus integers: 3600056 bytes

A generator object is a few hundred bytes whatever it will produce; the list pays for every element before you have seen one. Generator expressions — (x * 2 for x in items) — are lazy in the same way, where list comprehensions are eager. Materialise deliberately, when you need the values more than once or need them sorted.

yield from and infinite sequences

yield from delegates to a sub-generator, keeping composed generators readable: a function that does yield from inner() then yield 3 produces [1, 2, 3] when collected, exactly as if its two yields lived there. And because generators are lazy, they can represent sequences with no end at all — as long as you never ask for all of them.

import itertools

def naturals():
    n = 1
    while True:
        yield n
        n += 1
print(list(itertools.islice(naturals(), 5)))

Expected output: [1, 2, 3, 4, 5] — guarded by itertools.islice, which stops cleanly instead of hanging your terminal. The same code with list(naturals()) consumes memory until you interrupt it: the classic first mistake with infinite sequences.

A worked data pipeline

The whole chapter in one script: read a CSV lazily, filter it, count the survivors, and show what that approach costs in memory compared with loading everything.

# top_pages.py
import csv, sys
from pathlib import Path

def rows(path):
    with Path(path).open(newline="", encoding="utf-8") as f:
        yield from csv.DictReader(f)

def slow(rows_iter, minimum=30):
    for row in rows_iter:
        if int(row["seconds"]) >= minimum:
            yield row
tally = {}
for row in slow(rows("events.csv")):
    page = row["page"]
    tally[page] = tally.get(page, 0) + 1
for page in sorted(tally):
    print(f"{page}: {tally[page]}")
loaded = list(rows("events.csv"))
lazy = rows("events.csv")
total = sys.getsizeof(loaded) + sum(sys.getsizeof(row) for row in loaded)
print("list:", total, "bytes including", len(loaded), "rows")
print("generator:", sys.getsizeof(lazy), "bytes, no rows loaded yet")

Expected output: for a ten-row events.csv of page,seconds records:

/blog: 1
/contact: 1
/docs: 2
/pricing: 1
list: 2024 bytes including 10 rows
generator: 224 bytes, no rows loaded yet

The counter is deliberately blunt: for ten rows the difference is trivial, but the list grows with every row while the generator stays at 224 bytes — on a log file with ten million rows, only one of these approaches finishes.

Common mistakes

• Calling a generator function and expecting values — countdown(3) only builds a generator; the body has not run yet.
• Reusing an exhausted generator — it is one-shot, so the second list(gen) returns []; create the generator again.
• Implementing __next__ but not __iter__ — a for loop then refuses the object with TypeError: 'X' object is not iterable.
• Iterating an infinite generator without a bound — list(naturals()) never returns; use itertools.islice, a break or a sentinel.
• Materialising everything out of habit — sorted() or list() on a huge pipeline defeats the point; consume as far as you need, and no further.

Practice checklist

1. Write the Counter class by hand, iterate it, then delete __iter__ and read the TypeError.
2. Unroll any for loop from an earlier chapter into the while / try / except StopIteration form and confirm the output matches.
3. Add a print() inside a generator body, call the function, and note that nothing prints until you request the first value.
4. Build a three-stage pipeline over events.csv (read, filter, aggregate) and produce a report without ever calling list() on the whole file.
5. Feed an infinite generator to itertools.islice, take ten values, then try list() on it for one second and interrupt with Ctrl-C.

What's next?

Iteration and laziness are two of Python’s quiet superpowers; the third is the pattern you already used in chapter 15 without explanation — functions that wrap functions and with blocks that manage resources come from one idea. Start with Decorators and Context Managers.