Python Generators

Have you ever loaded a massive dataset into memory, only to have your program grind to a halt or crash with a MemoryError? If you've been using lists to process large streams of data — reading log files, iterating over API pagination, or generating infinite sequences — you're likely wasting memory and slowing down your code. Python generators solve this exactly: they produce items one at a time, on demand, without ever storing the whole sequence in memory. By the end of this lesson, you'll understand how to write generators, use generator expressions, and know exactly when they save you from memory headaches.

The problem this lesson solves

Imagine you need to process the first 10 million lines of a web server log file. If you read the entire file into a list, you'd need gigabytes of RAM just to hold the strings. Worse, you can't even start analyzing the data until the whole file is loaded. The same pain shows up when you want to generate an infinite sequence — like a stream of sensor readings or a never-ending Fibonacci series — because a list would never terminate. The root cause: eager evaluation builds the entire collection before you can use any element.

Python's classic pattern — for item in my_list — works fine for small collections. But when the data is large or unbounded, you need a tool that yields values lazily, only as you request them. That's exactly what generators provide.

Core concept / mental model

Think of a generator as a paused function with memory. When you call a regular function, it runs to completion and returns a single value. In contrast, a generator function yields one value, then pauses — saving all its local state, including the current line of execution — so it can resume later. The next time you ask for a value (via a for loop or next()), it picks up exactly where it left off.

Pro tip: Imagine you're reading a book one page at a time, then closing it and putting it on the shelf. When you come back, you open the book, find your bookmark, and keep reading. That's a generator. A list, on the other hand, would photocopy the entire book and hand you the stack of pages — memory heavy and slow.

A generator expression is the lightweight syntactic sibling — it looks like a list comprehension but uses parentheses () instead of brackets []. It produces a generator object, not a list.

Key definitions

• Generator function: any function that uses yield (instead of return). It returns a generator iterator.
• Generator iterator: the object produced by a generator function or expression. It supports iteration via __next__() and __iter__().
• Lazy evaluation: values are computed only when needed, not upfront.
• Memory efficient: because only one value exists at a time (before you overwrite it by requesting the next).

How it works step by step

Let's trace the lifecycle of a generator from call to exhaustion.

Step 1: Defining a generator function

def count_up_to(limit):
    num = 1
    while num <= limit:
        yield num
        num += 1

Notice yield instead of return. This function does not run when you call it — it returns a generator object.

Step 2: Creating the generator object

gen = count_up_to(5)
print(gen)          # <generator object count_up_to at 0x...>
print(type(gen))    # <class 'generator'>

At this point, no code inside the function has executed. The generator is in its initial paused state.

Step 3: Iterating (requesting values)

When you use for or next(), the function runs until the next yield statement, then pauses.

for value in gen:
    print(value)
# Output:
# 1
# 2
# 3
# 4
# 5

Each iteration: run up to yield, pause, return value, wait for next request.

Step 4: Exhaustion

After the last yield, the function returns (implicit return) — which raises StopIteration to signal the loop to stop. You cannot iterate over the same generator twice. To start over, you must create a new generator object.

Hands-on walkthrough

Let's build three practical examples that showcase generators in action.

Example 1: Read a large file line by line (memory-safe)

def read_large_file(file_path):
    with open(file_path) as f:
        for line in f:
            yield line.strip()

# Usage — never holds more than one line in memory
for line in read_large_file("huge_log.txt"):
    if "ERROR" in line:
        print(line)

Pro tip: This pattern is so common that file objects in Python are themselves iterators — you can just for line in open(file_path) — but the explicit generator demonstrates the concept and lets you add transformations (e.g., cleaning, filtering) in a lazy pipeline.

Example 2: Infinite Fibonacci sequence (never-ending stream)

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Grab the first 10 Fibonacci numbers on demand
fib = fibonacci()
first_10 = [next(fib) for _ in range(10)]
print(first_10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

No memory explosion — only one value at a time.

Example 3: Generator expression vs. list comprehension — memory comparison

import sys

# List comprehension — eager, stores all values
squares_list = [x * x for x in range(100_000)]
print(sys.getsizeof(squares_list))  # ~800,000+ bytes

# Generator expression — lazy, stores nothing except the expression
squares_gen = (x * x for x in range(100_000))
print(sys.getsizeof(squares_gen))   # ~112 bytes (tiny!)

# You can still iterate over the generator
for val in squares_gen:
    pass  # processes values one by one, no list overhead

Expected output (approximate per Python version):

824472
112

The list takes ~820 KB; the generator expression takes 112 bytes regardless of the range size. This is the power of lazy evaluation.

Compare options / when to choose what

Pro tip: If you need to iterate over data only once (e.g., to sum values, find a max, or write to a file), use a generator. If you need random access (e.g., my_list[5]), use a list — generators don't support indexing.

When to choose what

• Choose list comprehension when the sequence is small, you need to access elements by index, or you need to iterate multiple times.
• Choose generator expression for simple lazy transformations over large or streaming iterables — for example, sum(x*x for x in range(1_000_000)) uses almost no memory.
• Choose generator function when the iteration logic is complex (infinite sequences, maintaining state across yields, reading external resources, or implementing custom iterators).

Troubleshooting & edge cases

Mistake 1: Forgetting that a generator is single-use

gen = (i * 2 for i in range(5))
print(list(gen))  # [0, 2, 4, 6, 8]
print(list(gen))  # []  — generator is exhausted!

Fix: Create a new generator each time you need to iterate again, or convert to a list (defeating the memory benefit) if you need reuse.

Mistake 2: Using return with a value inside a generator

def my_gen():
    yield 1
    return "done"  # This line does NOT return data to the caller

A return inside a generator causes StopIteration to be raised. The return value is stored in StopIteration.value but is not accessible via for loops or next(). If you need to return a final result, consider using yield from or restructuring as an iterator class.

Mistake 3: Creating an infinite generator without a break condition in a loop

def endless():
    while True:
        yield 1

# This loop will run forever unless you break manually
for val in endless():
    pass  # Ctrl+C to stop

Fix: Either iterate over a limited range (for val in endless(): if some_condition: break) or consume only a fixed number of items using itertools.islice or a for loop with range.

Edge case: Generator with yield from

yield from delegates part of the iteration to another iterable or generator. It's syntactic sugar for nested loops over sub-generators.

def chain(*iterables):
    for it in iterables:
        yield from it

print(list(chain("abc", [1, 2, 3])))  # ['a', 'b', 'c', 1, 2, 3]

Edge case: Using .send() with generators (advanced)

Generators can receive values from the caller via .send(). This is an advanced pattern (coroutines) but powerful for two-way communication.

def counter():
    i = 0
    while True:
        new = yield i
        if new is not None:
            i = new
        else:
            i += 1

g = counter()
next(g)           # 0
g.send(10)        # 10
g.send(None)      # 11

What you learned & what's next

You now understand the core idea of generators: functions that can pause and resume, yielding values lazily to save memory. You've seen how to write generator functions with yield, use generator expressions for simple lazy sequences, and choose between lists, generator expressions, and generator functions based on your memory and access needs. You also learned common pitfalls like single-use exhaustion and infinite loops.

Key learning objectives achieved: - Explain the core idea behind generators: lazy evaluation via yield. - Complete a practical exercise: you wrote generators for file reading and infinite sequences.

What's next: The next lesson in this track will explore decorators with arguments — building on your understanding of functions to create flexible, reusable wrappers. You'll learn how to parameterize decorators to modify function behavior on the fly. The mental model of a generator's paused state will help you grasp how decorators wrap and manipulate functions.