You're Leaving CPU Cores on the Table: Here's How to Use Python's Multiprocessing

If you've ever watched your Python script chug through a massive dataset, fans spinning up but only one CPU core maxed out while others sit idle, you know the frustration. Python's Global Interpreter Lock (GIL) makes single-process threading mostly useless for CPU-bound tasks. But there's a solution that's been hiding in the standard library this whole time.

What Makes Multiprocessing Different

Unlike threading, which runs multiple tasks in the same memory space, multiprocessing spawns separate Python processes. Each gets its own GIL, its own memory space, and its own dedicated CPU core. The tradeoff? Memory overhead goes up, and inter-process communication gets more complicated.

But for number crunching, image processing, or any CPU-intensive work, it's a game changer.

Your First Multiprocessing Script

Let's start with something concrete. Say you're at PythonSkillset and you need to transform a list of prices from different currencies to USD. Here's the naive approach:

import time

prices = [100, 250, 75, 300, 150, 200, 125, 180]
exchange_rates = [0.85, 1.10, 0.73, 0.92, 1.05, 0.78, 0.95, 1.12]

def convert_to_usd(eur_price, rate):
    time.sleep(1)  # Simulated slow API call
    return eur_price * rate

start = time.time()
results = []
for price, rate in zip(prices, exchange_rates):
    results.append(convert_to_usd(price, rate))
print(f"Sequential: {time.time() - start:.2f}s")

That takes about 8 seconds with our simulated delay. Here's the multiprocessing version:

from multiprocessing import Pool

def convert_pair(pair):
    price, rate = pair
    return convert_to_usd(price, rate)

if __name__ == '__main__':
    start = time.time()
    with Pool(processes=4) as pool:
        results = pool.map(convert_pair, zip(prices, exchange_rates))
    print(f"Multiprocessing: {time.time() - start:.2f}s")

On a quad-core machine, that drops to roughly 2 seconds. Four processes, four cores, four times faster.

The Pool Object Is Your Best Friend

The Pool class handles the tedious work of distributing tasks and collecting results. Here are the methods you'll use most:

map() - Works like regular map but parallel. Perfect when each task is independent.

map_async() - Non-blocking version. Your main process can do other work while waiting.

apply_async() - For when you need different functions for different tasks.

starmap() - Like map but unpacks argument tuples. Great when your function takes multiple arguments.

Here's starmap in action:

def process_file(filename, directory, batch_size):
    # Do something expensive
    return f"Processed {filename}"

files = ["data1.csv", "data2.csv", "data3.csv"]
dirs = ["/raw", "/raw", "/clean"]
batches = [1000, 2000, 1500]

with Pool(processes=3) as pool:
    results = pool.starmap(process_file, zip(files, dirs, batches))

Sharing Data Between Processes

This is where beginners stumble. Each process has its own memory space, so you can't just use global variables. You need shared memory objects.

For simple values:

from multiprocessing import Value, Array

counter = Value('i', 0)  # 'i' for integer
shared_array = Array('d', [0.0, 1.0, 2.0])  # 'd' for double

def worker(counter):
    with counter.get_lock():
        counter.value += 1

For more complex data structures, use a Manager:

from multiprocessing import Manager

with Manager() as manager:
    shared_dict = manager.dict()
    shared_list = manager.list()

    # Works great for accumulating results
    def process_item(item):
        processed = expensive_operation(item)
        shared_list.append(processed)

Real World Example: Image Processing Pipeline

Let's say PythonSkillset needs to process 10,000 product images. Each image needs resizing, format conversion, and metadata extraction.

from multiprocessing import Pool, Manager
from PIL import Image
from pathlib import Path

def process_image(args):
    image_path, output_dir, results_list = args
    try:
        img = Image.open(image_path)
        img_resized = img.resize((800, 600))
        output_path = output_dir / f"processed_{image_path.name}"
        img_resized.save(output_path, "JPEG", quality=85)
        results_list.append({"original": str(image_path), "status": "success"})
    except Exception as e:
        results_list.append({"original": str(image_path), "status": "failed", "error": str(e)})

if __name__ == '__main__':
    image_dir = Path("/product_images")
    output_dir = Path("/processed_images")
    output_dir.mkdir(exist_ok=True)

    images = list(image_dir.glob("*.jpg"))

    with Manager() as manager:
        results = manager.list()
        with Pool(processes=8) as pool:
            tasks = [(img, output_dir, results) for img in images]
            pool.map(process_image, tasks)

        success_count = sum(1 for r in results if r["status"] == "success")
        print(f"Processed {success_count}/{len(images)} images")

When Multiprocessing Doesn't Help

Let's be honest about the limitations. Multiprocessing adds overhead for:

• Lightweight tasks - If each task takes microseconds, process spawning overhead dwarfs the work.
• IO-bound operations - File reads, API calls, database queries. Threading or asyncio works better here.
• Very large data transfer - Passing huge numpy arrays between processes eats up memory.

For these cases, consider concurrent.futures.ThreadPoolExecutor for IO tasks, or async frameworks like asyncio for network operations.

Common Pitfalls to Avoid

Forgetting if __name__ == '__main__': - Windows machines will spawn infinitely without this guard.

Passing non-pickleable objects - Multiprocessing pickles data to send between processes. Lambda functions, file handles, and many custom objects fail here.

Opening too many processes - multiprocessing.cpu_count() gives you the number of available cores. Going beyond that causes context switching overhead.

Ignoring error handling - If one process crashes, your entire pool goes down. Wrap your worker functions in try/except blocks.

Final Thoughts

The multiprocessing module won't solve every performance problem, but for CPU-bound work it's the standard library's hidden gem. Start with Pool.map(), use the if __name__ == '__main__' guard, and test with progressively more processes to find your sweet spot.

Your CPU has been waiting for you to use all those cores. Give it a reason to work.