Multiprocessing for Parallel Tasks

You spend hours writing Python scripts that process large datasets, transform images, or compute heavy mathematical models — but they still run on a single CPU core, leaving the other 7, 15, or 63 cores completely idle. The pain is real: your program takes minutes when it could take seconds. Python's multiprocessing module is the key to unlocking that hidden horsepower by running tasks truly in parallel across multiple CPU cores, bypassing the infamous Global Interpreter Lock (GIL). This lesson will teach you how to use Process, Pool, and Queue to turn your CPU‑bound Python code into a parallel powerhouse.

The problem this lesson solves

Python's Global Interpreter Lock (GIL) is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. This means that threading — while great for I/O‑bound tasks — does not let you run CPU‑intensive work in parallel on multiple cores. The result? Your multi‑threaded code for number crunching or data processing often runs slower than its single‑threaded counterpart due to the overhead of context switching.

Pro Tip: If you are doing I/O (file reads, network requests, database queries), threading or asyncio is the solution. For CPU‑bound work (math, image processing, file compression) — you need multiprocessing.

The real‑world cost

Imagine you have 10,000 high‑resolution images to resize. Using a single thread on a 4‑core machine, it might take 60 seconds. With multiprocessing, you can reduce that to ~15 seconds — a 4× speedup — because each core handles a batch of images simultaneously. No GIL to block you.

Core concept / mental model

Think of your computer's CPU as a factory floor: one core is one worker. Normally, Python uses only one worker (the main process) no matter how many workers are available. multiprocessing lets you hire multiple workers (separate processes), each with their own memory space, and each can run Python code in parallel without fighting over the GIL.

Key definitions

• Process: An independent program in memory — has its own Python interpreter, memory, and GIL. Can run on a separate CPU core.
• Pool: A convenient way to manage a bunch of worker processes. You give it a function and a list of inputs; it distributes the work across workers.
• Queue: A thread‑ and process‑safe way to pass data between processes (like messages on a conveyor belt).
• GIL: Global Interpreter Lock — the single lock that multiprocessing sidesteps by using separate processes.

How processes differ from threads

How it works step by step

Here is the mental flow for parallelizing a task with multiprocessing:

1. Identify the CPU‑bound function — what heavy work is being repeated many times?
2. Create a Pool of workers — decide how many processes you want (default: number of CPU cores).
3. Map the function over your data — like map() but each element is processed by a different worker in parallel.
4. Collect the results — the pool returns a list of outputs once all workers finish.
5. Close and join — clean up when done.

The anatomy of Pool.map()

import multiprocessing as mp

def square(x):
    return x * x

if __name__ == "__main__":
    with mp.Pool(processes=4) as pool:
        results = pool.map(square, range(10))
    print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Critical: Always guard your main module with if __name__ == "__main__": on Windows to prevent infinite process spawning. On Unix it's optional but good practice.

Hands-on walkthrough

Let's parallelize a CPU‑intensive task: computing the sum of squares for a range of numbers. First, the single‑core baseline:

Example 1: Single‑core version

import time

def cpu_intensive(n):
    total = 0
    for i in range(n):
        total += i ** 2
    return total

if __name__ == "__main__":
    t0 = time.perf_counter()
    results = [cpu_intensive(20_000_000) for _ in range(4)]
    t1 = time.perf_counter()
    print(f"Sequential took {t1 - t0:.2f} seconds")

Expected output (example):

Sequential took 2.85 seconds

Example 2: Multiprocessing with Pool

import multiprocessing as mp
import time

def cpu_intensive(n):
    total = 0
    for i in range(n):
        total += i ** 2
    return total

if __name__ == "__main__":
    t0 = time.perf_counter()
    with mp.Pool(processes=4) as pool:
        results = pool.map(cpu_intensive, [20_000_000] * 4)
    t1 = time.perf_counter()
    print(f"Parallel with 4 processes took {t1 - t0:.2f} seconds")

Expected output (example on 4‑core machine):

Parallel with 4 processes took 0.78 seconds

That's about 3.6× faster — and close to the theoretical 4× speedup.

Example 3: Using Process directly when you need fine control

Sometimes a Pool is too high‑level. If you need to run different functions or have separate control flows, you can spawn individual Process objects and use a Queue to collect results:

import multiprocessing as mp
import time

def worker(n, q):
    total = 0
    for i in range(n):
        total += i ** 2
    q.put(total)

if __name__ == "__main__":
    q = mp.Queue()
    processes = []
    for _ in range(4):
        p = mp.Process(target=worker, args=(20_000_000, q))
        processes.append(p)
        p.start()

    results = [q.get() for _ in range(4)]
    for p in processes:
        p.join()
    print(f"Results: {results}")

Expected output:

Results: [2666666666666667, 2666666666666667, 2666666666666667, 2666666666666667]

Why use Process? When you need to run different functions, have custom lifecycles, or share state via Manager or Value.

Compare options / when to choose what

Pro Tip: For 90% of CPU‑bound parallel tasks, mp.Pool.map() or mp.Pool.starmap() is all you need. Reach for Process only when you have different functions or custom inter‑process communication.

Troubleshooting & edge cases

1. "RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping."

This happens when you forget the if __name__ == '__main__': guard on Windows.

Fix: Always wrap the code that creates Process or Pool inside if __name__ == '__main__':.

2. The script runs but only 1 core is used (no speedup)

You might be accidentally using map twice (once to create the pool, once in the function) or the task is actually I/O‑bound. Also, check if your function has a tight loop that is too short for the overhead to be worthwhile.

Fix: Use time.perf_counter() to measure and verify the speedup. If you see no improvement, increase the task size or use starmap with larger data.

3. Processes hang or never finish

This often happens with a Queue that overflows when you don't get() results quickly enough, or when a child process tries to share mutable objects without a Manager.

Fix: Use Queue with a maxsize parameter, or use Manager.list() / Manager.dict() for shared state. Ensure every put() has a matching get().

4. Process still not releasing the GIL?

You might be confusing threads with processes. With multiprocessing, each process has its own GIL, so the GIL is not an issue. But if you mix multiprocessing with third‑party C extensions (like numpy) that release the GIL internally, be careful — they might still have their own internal locks.

5. Memory explosion

Each child process has its own memory space, so if you pass large data to each worker, memory usage multiplies. For huge datasets, consider shared memory via multiprocessing.Array or multiprocessing.shared_memory (Python 3.8+).

What you learned & what's next

You now understand how to break free from the GIL using Python's multiprocessing module: - What the GIL is and why threading fails for CPU‑bound work. - How to create a Pool of worker processes and use pool.map() for automatic parallel execution. - When to reach for Process + Queue for greater control. - Real‑world pitfalls: if __name__ guard, queue deadlocks, and memory scaling.

With this in your toolkit, you can parallelize image processing, data transformations, scientific simulations, and batch file operations — often achieving near‑linear speedups on multi‑core machines.

Next step: In the next lesson, you'll learn about inter‑process communication with Pipe and Manager — essential for building more complex parallel workflows where processes need to share state or synchronize. Continue your journey toward mastering concurrent Python!