Yield from Generators for Streams

Imagine you're processing a massive log file that doesn't fit into memory — or piping data from an API that sends gigabytes of JSON. Manually chaining generators with nested loops or list conversions leads to memory bloat, unreadable code, and bugs. Python's yield from syntax solves this elegantly, letting you delegate iteration to another generator while keeping your pipeline lazy and composable. It's the secret weapon for building data streams that are both memory-efficient and clean.

The problem this lesson solves

When you have multiple generators that need to feed into each other, the naive approach is either to collect all results into a list (wasting memory) or to write awkward for item in sub_generator: yield item loops. For real-world data streams — like processing sensor readings, paginated API responses, or multi-stage ETL pipelines — these patterns become repetitive and error-prone.

Consider this: You built a generator that reads a CSV file line by line, another that filters rows, and a third that transforms columns. Without yield from, you'd nest them like this:

def read_data():
    for row in csv_reader:
        yield row

def filter_positive(data_gen):
    for row in data_gen:
        if row['value'] > 0:
            yield row

This works, but every extra layer adds a for loop and a yield — code duplication that hides the intent of your pipeline. Worse, if you need to merge multiple streams (e.g., combine data from two files), you end up with even more boilerplate.

Pro Tip: yield from reduces this boilerplate to a single line, making the data flow obvious to anyone reading your code.

Core concept / mental model

Think of yield from as a generator delegation operator. When a generator uses yield from other_gen, it says: "Run other_gen completely, and pass every value it produces directly to the consumer (the outer for loop or next() call)." This is like a conductor handing the baton to a soloist — the audience hears the music without noticing the handoff.

Key definitions: - Generator: A function with yield that produces a sequence lazily. - Delegating generator: The outer generator that uses yield from. - Subgenerator: The inner generator that is delegated to.

Diagram in words:

Consumer (for loop)
    ↑
Delegating Generator (uses yield from)
    ↑
Subgenerator 1 (produces values)
    ↑
(optional) Subgenerator 2

yield from also handles return values from subgenerators: you can capture the final return value of the subgenerator if it uses return some_value. This is perfect for scenarios where a subgenerator computes a summary (like a count or sum).

How it works step by step

1. Without yield from (manual delegation)

def counter(max):
    i = 0
    while i < max:
        yield i
        i += 1

def chain_slow(gen1, gen2):
    # Manual forwarding
    for item in gen1:
        yield item
    for item in gen2:
        yield item

2. With yield from (clean delegation)

def counter(max):
    i = 0
    while i < max:
        yield i
        i += 1

def chain_fast(delegates):
    # Delegating to multiple generators
    for gen in delegates:
        yield from gen

result = list(chain_fast([counter(3), counter(2)]))
print(result)  # Output: [0, 1, 2, 0, 1]

How it works internally: 1. The outer generator chain_fast is called. 2. It iterates over the list of generators. 3. For each generator, yield from pauses the outer generator and starts the subgenerator. 4. Every value yielded by the subgenerator is directly passed to the consumer. 5. When the subgenerator is exhausted, control returns to the outer generator.

3. Capturing return values from subgenerators

def accumulate(data):
    total = 0
    for val in data:
        yield val * 2
        total += val
    return total

def pipeline(data):
    result = yield from accumulate(data)
    print(f"Subgenerator returned: {result}")  # Side effect
    # You can also yield the return value if needed
    yield result

data = [1, 2, 3]
gen = pipeline(data)
for item in gen:
    print(item)
# Output:
# 2
# 4
# 6
# Subgenerator returned: 6
# 6

Here, accumulate returns a total after yielding all transformed values. The yield from expression evaluates to the return value of the subgenerator. This is incredibly useful for aggregating partial results in data streams.

Hands-on walkthrough

Let's build a real-world data stream that reads numbers from two sources, filters them, and computes a running sum.

Step 1: Create the generators

def number_source(name, count):
    """Simulates a stream of numbers from a source."""
    for i in range(count):
        yield f"{name}:{i}"

def parse_and_filter(source_gen):
    """Parses string, keeps only even numbers."""
    for item in source_gen:
        _, num_str = item.split(':')
        num = int(num_str)
        if num % 2 == 0:
            yield num

def running_sum(source_gen):
    """Yields each number and accumulates a running total."""
    total = 0
    for val in source_gen:
        total += val
        yield val, total
    return total

Step 2: Compose the pipeline

def compose_pipeline():
    source1 = number_source('A', 5)
    source2 = number_source('B', 5)

    # Merge both sources with yield from
    def merged():
        yield from source1
        yield from source2

    filtered = parse_and_filter(merged())
    result = running_sum(filtered)

    for val, accum in result:
        print(f"Added {val}, running total: {accum}")

    # To get the final return value from running_sum, we need a different approach:
    # We can wrap it
    def wrapper():
        return yield from running_sum(filtered)

compose_pipeline()
# Output:
# Added 0, running total: 0
# Added 2, running total: 2
# Added 4, running total: 6
# Added 0, running total: 6
# Added 2, running total: 8
# Added 4, running total: 12

Step 3: Use yield from to capture final aggregate

def pipeline_with_aggregate(data_sources):
    total = 0
    for num in data_sources:
        # Delegate to a subgenerator that aggregates
        result = yield from some_aggregator(num)
        total += result
    return total

Blockquote: If you forget that yield from can return a value, you'll miss the elegance of aggregator patterns. Always check if your subgenerator uses return to finalize.

Compare options / when to choose what

When to choose yield from: - You need to chain multiple generators cleanly. - You want to capture return values from subgenerators. - You're building reusable data pipeline components.

When to use alternatives: - Use itertools.chain for simple merging without transformation (no custom subgenerator logic). - Use manual for loops when you need fine-grained control between generator iterations. - Never use lists for large data streams.

Troubleshooting & edge cases

Common Mistake 1: yield from with a plain iterator that has no return value

If the subgenerator simply yields values and never uses return, yield from returns None. This is usually fine — just don't assign the result if you don't need it.

Common Mistake 2: Attempting to use yield from outside a generator function

yield from is only valid inside a generator (a function that contains yield or yield from). Using it in a regular function raises SyntaxError.

Common Mistake 3: Forgetting that yield from exhausts the subgenerator completely

Once you delegate, you cannot iterate over the subgenerator again. If you need to reuse data, wrap it in a list (but then you lose memory efficiency).

Edge case: Subgenerator returning None vs. raising StopIteration with a value

Python 3.7+ raises StopIteration with a value if the subgenerator uses return value. yield from captures that value correctly. In older Python versions, the behavior was problematic — ensure you're on 3.10+ for latest features.

Edge case: Generators with send() and throw()

yield from fully delegates the bidirectional communication. If you send() a value to the delegating generator, it goes to the subgenerator. This is advanced but powerful for co-routine patterns.

What you learned & what's next

You've mastered yield from for data streams — a critical skill for handling large datasets without memory explosion. You now know: - How to delegate iteration from one generator to another with a single line of code. - How to capture return values from subgenerators for aggregations. - How to compose multi-source pipelines that are readable and efficient.

Your next step: Apply this to build a generator that reads a multi-gigabyte CSV, filters rows with a condition, transforms columns, and yields only the needed output. Then, explore asyncio generators for streaming data concurrently.

Remember: Every time you write for item in gen: yield item, ask yourself: "Can I replace this with yield from gen?" The answer is almost always yes, making your code cleaner and faster.