Batch Process Files with Python's Multiprocessing

How to Batch Process Files with Python's Multiprocessing

Have you ever sat there staring at a progress bar crawling along while Python processes files one by one? Yeah, we've all been there. It's like watching paint dry, except the paint is your time and it's literally evaporating.

Here's the thing: modern computers have multiple cores sitting mostly idle while your script stubbornly processes files in a single thread. That's just wasteful.

The Problem With Sequential Processing

Let's look at a common scenario. Say you have 10,000 images that need resizing, or a million log files that need parsing. The naive approach looks something like this:

import glob
import time

def process_file(filename):
    # Simulate some CPU-intensive work
    time.sleep(0.1)
    return f"Processed {filename}"

if __name__ == "__main__":
    files = glob.glob("*.txt")
    start = time.time()

    for f in files:
        result = process_file(f)

    print(f"Took {time.time() - start} seconds")

With 100 files, that's 10 seconds. With 10,000 files, you'd better grab a coffee. Maybe two.

Your First Multiprocessing Script

The multiprocessing module in Python's standard library makes parallel processing surprisingly simple. Here's the same task, but now using all your CPU cores:

import glob
import time
from multiprocessing import Pool

def process_file(filename):
    # Same function as before
    time.sleep(0.1)
    return f"Processed {filename}"

if __name__ == "__main__":
    files = glob.glob("*.txt")
    start = time.time()

    with Pool() as pool:
        results = pool.map(process_file, files)

    print(f"Took {time.time() - start} seconds")

The magic happens in Pool(). By default, it creates one worker process per CPU core. So on a typical quad-core machine, that 10 second job now takes about 2.5 seconds.

Understanding the Pool

The Pool object is your work dispatcher. It takes your list of inputs and distributes them across worker processes. The map() method is the star player here — it's basically the same as Python's built-in map(), but parallel.

You can also control how many workers you want:

# Use exactly 4 workers
with Pool(processes=4) as pool:
    results = pool.map(process_file, files)

Handling Different Input/Output Patterns

Sometimes your function needs more than just a filename. Maybe you need to pass a configuration dictionary:

from functools import partial
from multiprocessing import Pool

def process_file_with_config(filename, config):
    # Use config settings here
    return f"Processed {filename} with config"

if __name__ == "__main__":
    files = glob.glob("*.txt")
    config = {"quality": 85, "format": "jpeg"}

    with Pool() as pool:
        func = partial(process_file_with_config, config=config)
        results = pool.map(func, files)

Real-World Example: Image Resizing

Let's say Pythonskillset has a folder of 10,000 product photos that need resizing. Here's how you'd actually do it:

from PIL import Image
from multiprocessing import Pool
import glob
import os

def resize_image(filename):
    output_dir = "resized"
    os.makedirs(output_dir, exist_ok=True)

    with Image.open(filename) as img:
        img_resized = img.resize((800, 600))
        output_path = os.path.join(output_dir, filename)
        img_resized.save(output_path)

    return f"Resized {filename}"

if __name__ == "__main__":
    images = glob.glob("*.jpg")

    with Pool() as pool:
        results = pool.map(resize_image, images)

    print(f"Resized {len(results)} images")

Common Pitfalls to Avoid

Memory consumption: Each worker process gets its own memory space. If each processed file takes 500MB, running 8 workers means 4GB of RAM usage. Dial it back if needed:

with Pool(processes=2) as pool:  # Be conservative with memory
    results = pool.map(process_large_file, files)

Windows compatibility: On Windows, your main script must be protected by if __name__ == "__main__". This isn't optional — without it, Windows will crash trying to spawn child processes.

Exception handling: If one worker fails, the entire pool shuts down. Catch exceptions within your worker function:

def safe_process(filename):
    try:
        # Your processing logic
        return ("success", process_file(filename))
    except Exception as e:
        return ("error", f"{filename}: {str(e)}")

When Multiprocessing Doesn't Help

Here's the honest truth: multiprocessing only speeds up CPU-bound tasks. If your bottleneck is reading from a slow disk or waiting for network responses, multiple processes won't help much. Those are IO-bound tasks, and you'd be better off with asyncio or threading.

The Takeaway

Batch processing files with multiprocessing is one of those rare cases where a simple change yields immediate, dramatic speed improvements. The core insight is beautiful in its simplicity: instead of making one worker do everything, let all your CPU cores share the load.

Next time you find yourself waiting for a file processing script, remember: your computer has more to give. All it takes is a few lines of code to unlock it.

Start with Pool().map() and watch your scripts fly.