PY-01-018-P · Primer — Iterators & Generators

The idea in one sentence

An iterator is any object that hands you its values one at a time — you ask for the next one with next() and it eventually says "I'm done" — and a generator is the effortless way to build one: an ordinary-looking function that uses yield instead of return to produce values lazily, pausing between each so you never have to hold the whole sequence in memory at once.

Why it exists

Every for loop you've ever written secretly relies on this machinery, and once you see it you can build sequences that would otherwise be impossible. The problem it solves is simple but everywhere: sometimes you want to walk through a sequence of values without building the whole thing first. Reading a ten-gigabyte log file into a list would crash your machine; generating the first prime after a billion, or an endless stream of sensor readings, can't be a list at all because it never ends. Python's answer is lazy evaluation — compute each value only at the moment it's asked for, then forget it and move on — and the iterator protocol is the tiny, uniform contract that makes this work across lists, files, dictionaries, and your own custom objects. Generators exist because writing that protocol by hand is tedious; yield lets you describe "produce this, then this, then this" as a normal function, and Python turns it into a memory-light, pause-and-resume iterator for free. The payoff is code that reads like it builds a list but runs like it barely uses memory.

How it works

The iterator protocol has just two moves: call iter(obj) to get an iterator, then call next(it) repeatedly to pull values until it raises StopIteration. A generator function is any def that contains yield; calling it doesn't run the body — it hands back a generator object that is already an iterator. Each next() runs the body up to the next yield, returns that value, and freezes the function's local state right there; the following next() thaws it and continues. Because the work happens one step at a time, nothing is precomputed and only one value lives in memory at a time. A for loop does this whole dance for you — iter, repeated next, catch StopIteration — so a generator drops straight into any loop.

def countdown(n):
    print("countdown starts")
    while n > 0:
        yield n          # hand back one value, then PAUSE right here
        n -= 1
    print("countdown done")

gen = countdown(3)       # nothing prints yet -- the body has not run
print(next(gen))         # countdown starts   (then) 3
print(next(gen))         # 2
print(next(gen))         # 1

for x in countdown(2):   # a for loop drives next() for you
    print("tick", x)     # countdown starts / tick 2 / tick 1 / countdown done

Notice that "countdown starts" prints only when the first value is pulled — proof the function is genuinely lazy.

The one gotcha

An iterator is one-shot: once it's exhausted, it stays exhausted — looping again gives you nothing. A list you can iterate a hundred times, but a generator (and any iterator) yields its values exactly once and then is spent forever. This bites people who store a generator and reuse it:

squares = (n * n for n in range(3))   # a generator expression
print(list(squares))   # [0, 1, 4]   <- first pass drains it
print(list(squares))   # []          <- nothing left; it's exhausted
print(sum(squares))    # 0           <- still empty

The fix is a mindset shift: treat a generator as a stream you consume once, not a container you revisit. If you truly need the values twice, either materialize them with list(...) (accepting the memory cost) or call the generator function again to get a fresh iterator. And if you drive next() by hand past the end, be ready to catch StopIteration — that exception isn't an error, it's the iterator's normal way of announcing "that's all."

See it — how one next() call drives a generator

flowchart TD
    A["for x in countdown(2)"] --> B["iter(...) gets the<br/>generator object"]
    B --> C["call next()"]
    C --> D["run body until<br/>the next yield"]
    D --> E{"Reached a<br/>yield?"}
    E -->|"Yes"| F["return that value,<br/>FREEZE local state"]
    F --> G["loop body uses x"]
    G --> C
    E -->|"No: fell off<br/>the end"| H["raise StopIteration"]
    H --> I["for loop stops<br/>cleanly"]

The one-line takeaway

Use a generator — a function with yield — to produce values lazily one at a time, so any sequence, even an endless or file-sized one, flows through a for loop without ever living in memory all at once.

Where you'll meet it

You already meet iterators every time you write for line in open("data.txt"): — a file is an iterator that streams one line at a time, which is why huge files loop fine. You'll reach for generators the moment "build a list, then loop over it" feels wasteful: (row for row in reader if row.active) filters a million records while holding one, and a def read_batches(): that yields chunks lets you process a dataset far bigger than RAM. The whole itertools module is generators (count, chain, islice), range and zip and enumerate are lazy for the same reason, and pipelines that chain generator into generator are a hallmark of clean, scalable Python. Once the pause-and-resume picture clicks, you'll start writing data flows that stay tiny in memory no matter how big the input grows — a quiet superpower that separates scripts that crash on real data from ones that just keep humming.

Named concepts: the iterator protocol, iter and next, generator functions with yield, lazy evaluation and memory efficiency