Generators: a function you can pause

yield turns a function into a resumable state machine, so it can produce a huge sequence in constant memory.

Transcript

The Python generator types in while a cursor runs through the function body, stops at yield to hand out a value, and resumes from the same line on the next call while memory stays one frame.

def count_up(n):
    print("start")
    i = 0
    while i < n:
        yield i
        print("resumed after", i)
        i += 1

g = count_up(3)
print("created")
print(next(g))
print(next(g))

It prints:

created
start
0
resumed after 0
1

Why#

Calling a function that contains yield runs none of its body. Python returns a generator object that owns a suspended frame: the local variables plus the position of the next instruction. Each next(g) resumes that frame, runs to the next yield, hands the value out and freezes again. That is why start prints after created, and why resumed after 1 never prints: nobody asked for a third value.

The state is one frame, not the sequence, so memory does not grow with n. sys.getsizeof(count_up(10)) and sys.getsizeof(count_up(10**12)) report the same size, while list(range(10**6)) is 8 MB of pointers before counting the integers.

You can watch the machine: inspect.getgeneratorstate(g) moves through GEN_CREATED, GEN_SUSPENDED and GEN_CLOSED.

Where it breaks#

• One pass only. An exhausted generator is not an error, it is empty: list(g) twice gives the values, then []. Silent bugs live here.
• No len(), no indexing. If you need them, build the list and pay for it.
• return x ends a generator and x rides out on StopIteration.value. Since Python 3.7 (PEP 479), a StopIteration raised inside the body becomes RuntimeError: generator raised StopIteration, so a stray next() cannot end your loop by accident.

Rule of thumb#

If you loop over it once, yield instead of append. A generator is a perfect procrastinator: it does exactly the work you ask for, exactly when you ask, and not one line sooner.