Generator Functions

You have a function that returns a list — say, a list of the first million Fibonacci numbers. It works, but your program grinds to a halt, consuming gigabytes of RAM before you can even use the first number. The core issue isn't the algorithm; it's when the function computes those values: it builds the entire list in memory before returning. This is exactly the pain that generator functions in Python eliminate. They allow you to produce items one at a time, on demand, transforming huge data streams into a lightweight, memory-efficient flow.

The problem this lesson solves

When you use a regular function to return a collection, you pay the full price upfront. A function that return sum([i**2 for i in range(10**7)]) first builds a 10-million-element list in memory, then sums it. For massive datasets, log files, or infinite sequences, this approach is a showstopper. It wastes memory, slows down your program with large allocations, and often processes data you don't even need.

Generator functions are the antidote. They let you write code that yields one value at a time, pausing execution between yields. Instead of a complete list, you get an iterator that produces values lazily — only when your loop or consumer asks for the next item. This changes memory usage from O(n) to O(1) for iteration, and can make previously impossible tasks (like streaming a log file of hundreds of GB) straightforward.

Core concept / mental model

Think of a generator function not as a factory that stamps out a full batch of items, but as a conveyor belt operator who places items on the belt one by one, only as the next worker (your code) picks one up. Each yield is like the operator placing a single item on the belt and then taking a break. Your code grabs it, does something, and asks for the next. The operator has no memory of previous items once they are taken — only the current position and state.

Pro tip: Every generator function is a suspended computation. When you call it, it returns an iterator object. The function body does not start running until you call next() on that iterator. This is fundamentally different from a regular function, which runs from start to finish immediately.

Definition

• Generator function: A function that contains at least one yield statement. It returns a generator iterator object.
• Generator iterator: The object produced by calling a generator function. It follows the iterator protocol (__iter__ and __next__), meaning you can use it in for loops, list() constructors, and anywhere an iterable is accepted.
• Lazy evaluation: Values are computed only when requested, not ahead of time.

How it works step by step

Let's trace the lifecycle of a generator function call:

1. Define: You write a function using def and yield instead of return.
2. Call: Calling the function does not execute its body. It returns a generator object. You can inspect it: it's an iterator.
3. First next() call: The function body begins executing from line 1. It runs until it hits a yield statement. The yielded value is returned to the caller. Execution pauses at that point, preserving all local variables and the instruction pointer.
4. Subsequent next() calls: Execution resumes right after the yield. It continues until the next yield, or until the function naturally ends or raises StopIteration.
5. Exhaustion: When the function returns (even implicitly), the generator raises StopIteration. You cannot get more values.

Example: A simple counter

def count_up_to(n):
    count = 1
    while count <= n:
        yield count        # pause and send count to caller
        count += 1         # resume here on next next()

# Usage
counter = count_up_to(3)
print(next(counter))  # 1
print(next(counter))  # 2
print(next(counter))  # 3
# print(next(counter))  # raises StopIteration

Manual next() vs. for loop

Using a for loop is safer and more Pythonic — it handles StopIteration and cleans up the generator:

def even_numbers(limit):
    for num in range(limit):
        if num % 2 == 0:
            yield num

for even in even_numbers(10):
    print(even)
# Output: 0 2 4 6 8

Hands-on walkthrough

We'll build a generator that reads a large file line by line, processes it, and yields results — a common real-world task that benefits hugely from lazy evaluation.

1. Generator that counts words in lines

Assume we have a file data.txt with one line per record. We want to yield the number of words per line, without loading the whole file.

def word_count_per_line(filename):
    """Yield word count for each line in a file."""
    with open(filename, 'r', encoding='utf-8') as file:
        for line in file:
            yield len(line.split())

# Usage (file has 4 lines with varying words)
counts = word_count_per_line('data.txt')
for cnt in counts:
    print(cnt)

If data.txt contains:

hello world
test
foo bar baz
x

Output:

2
1
3
1

2. Generator that yields processed lines

Now let's create a generator that yields only lines that contain a specific keyword, transforming them to uppercase.

def keyword_lines(filename, keyword):
    """Yield lines containing keyword, in uppercase."""
    with open(filename, 'r') as file:
        for line in file:
            if keyword in line:
                yield line.strip().upper()

# Usage
for line in keyword_lines('data.txt', 'world'):
    print(line)
# Output: HELLO WORLD

3. Infinite generator (use with caution)

Generators can model infinite sequences — useful for streams, counters, or simulation.

def fibonacci_infinite():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Get first 10 Fibonacci numbers
fib = fibonacci_infinite()
first_10 = [next(fib) for _ in range(10)]
print(first_10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Pro tip: When consuming an infinite generator, always have a way to stop — like a break condition or itertools.islice. Otherwise your loop runs forever.

Compare options / when to choose what

When to choose a generator function

• You are processing data that cannot fit in memory (e.g., a 50 GB CSV file).
• You need a pipeline of one-time transformations (e.g., generator chaining).
• You want to model an infinite stream (sensor data, UI events).
• You are building a custom iterator that maintains state efficiently.

When to stick with lists

• You need the data multiple times (generators are exhausted after one loop).
• The dataset is small and you need random access (list[3]).
• You need to modify elements (generators don't support item assignment).

Troubleshooting & edge cases

1. Generator exhausted too early

Symptom: A generator yields correctly the first time, but the second loop over the same variable produces nothing.

Why: A generator object can be iterated only once. After StopIteration, it's done.

def simple():
    yield 1
    yield 2

gen = simple()
print(list(gen))  # [1, 2]
print(list(gen))  # []  -> exhausted!

Fix: Create a new generator object for each full iteration: list(simple()) twice is fine.

2. yield vs. return: mixing them

Symptom: A function that uses both yield and return may confuse new learners.

Rule: If a function has any yield, it is a generator. A return inside a generator is not illegal — it raises StopIteration and the returned value is attached to it (available via StopIteration.value). But for most purposes, simply let the generator fall off the end.

def bad_mix():
    yield 1
    return "done"   # Not recommended; catches beginners off guard

val = next(bad_mix())
print(val)  # 1
# next(bad_mix())  # StopIteration raised; returned value hidden

Fix: Use return without a value to stop early, or just avoid mixing. Prefer yield from for delegation.

3. Infinite loop without escape

Symptom: A generator with while True: yield something runs forever when used in a for loop with no break.

Fix: Always think about the consumer. Use itertools.islice(), a counter, or embed a break condition.

import itertools

def endless_numbers():
    i = 0
    while True:
        yield i
        i += 1

# Safe use: take first 5
for num in itertools.islice(endless_numbers(), 5):
    print(num)  # 0 1 2 3 4

4. Generator not consumed — no side effects

Symptom: You call a generator function expecting it to print something, but nothing happens.

Reason: No next() call means zero lines of the function run.

def lazy_printer():
    print("I only run when iterated")
    yield 42

lazy_printer()  # No output!

Fix: Iterate through it: list(lazy_printer()) or for x in lazy_printer(): pass.

What you learned & what's next

You now understand how generator functions solve the memory/performance pain of large or infinite data sequences. You've learned to: - Write generators using yield instead of return. - Grasp the lazy evaluation mental model: paused execution, state preservation, one item at a time. - Apply them to file processing, infinite sequences, and efficient pipelines. - Compare generators with lists and know when to choose each. - Avoid common gotchas like exhaustion, mixing return, and infinite loops.

What's next: In the next lesson, you'll learn generator expressions — the concise, list-comprehension-like syntax for creating generators on the fly. They combine the memory efficiency of yield with the brevity of comprehensions. After that, you'll explore yield from and advanced generator patterns like coroutines.

Practice recap: Write a generator function prime_numbers() that yields all prime numbers up to a given limit, one at a time. Then use it to sum the first 20 primes. Compare the memory usage with an equivalent list-based version. Try processing a large text file (e.g., a novel) and yielding the lines that contain the word 'the' — see how little memory it uses.