Multiprocessing — True Parallelism for CPU-Bound Work
~18 min · multiprocessing, process, parallel, executor
Level 0Curious
0 XP0/93 lessons0/23 achievements
0/100 XP to next level100 XP to go0% complete
Why multiprocessing exists
The GIL prevents two threads from running Python bytecode in parallel. Multiprocessing sidesteps this by using separate Python processes — each with its own GIL, its own memory, its own everything. For CPU-bound work that you want to actually parallelize, this is the answer.
The cost — process startup and serialization
Starting a process is far slower than starting a thread (milliseconds vs microseconds). Passing data between processes requires serialization — Python objects pickle/unpickle across the boundary. So multiprocessing wins when the work-per-task is bigger than the startup + communication overhead. For a 1-second computation, the overhead is negligible. For a 1ms task, threading or no concurrency is faster.
ProcessPoolExecutor — the easy way
concurrent.futures.ProcessPoolExecutor has the same API as ThreadPoolExecutor. Just swap the class. The function and arguments must be picklable, which means: defined at the top level (not inside another function), the arguments must be standard types (not lambdas, not file handles, etc.). This catches people the first time.
Sharing state — Manager and shared memory
Processes don't share memory by default. multiprocessing.Manager gives you proxies to shared lists/dicts that work across processes (with overhead). multiprocessing.shared_memory (3.8+) gives raw shared memory blocks for high-throughput numeric data. Most multiprocessing code doesn't need these — pass arguments in, return results out, no shared state.
Principle: Multiprocessing for CPU-bound. Asyncio (or threading) for I/O-bound. The wrong choice is the most common production mistake — like running an HTTP server with multiprocessing or a CPU-heavy computation in asyncio.
Values crossing a process boundary are copied or serialized. Sending a huge object for every tiny task can cost more than the computation itself. Keep inputs compact, make work units substantial, and return only the result the parent actually needs.
Code
ProcessPoolExecutor — same API, different model·python
from concurrent.futures import ProcessPoolExecutor
import time
def compute_squares(n):
return sum(i * i for i in range(n))
if __name__ == "__main__": # required for multiprocessing on some platforms
start = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as pool:
results = list(pool.map(compute_squares, [1_000_000] * 4))
print(f"results count: {len(results)}")
print(f"elapsed: {time.perf_counter() - start:.2f}s")
# With 4 processes on a multi-core machine, runs ~4x faster
# than the same in a single process or with ThreadPoolExecutor
# (because of the GIL)
if __name__ == '__main__': — why it matters·python
# On Windows and macOS-spawn, multiprocessing imports the script
# in each worker. Without the if __name__ guard, child workers
# would re-execute the multiprocessing setup and recursively spawn.
#
# Always guard top-level multiprocessing code:
#
# def work(x):
# return x * 2
#
# if __name__ == '__main__':
# with ProcessPoolExecutor() as pool:
# results = list(pool.map(work, range(10)))
# print(results)
# Functions must be picklable — defined at module top level
Pickling — what works, what doesn't·python
from concurrent.futures import ProcessPoolExecutor
# This works — module-level function, picklable args
def double(x):
return x * 2
# This DOESN'T work — lambda is not picklable
# squarer = lambda x: x * x
# pool.submit(squarer, 5) # PicklingError
# Workaround — define the function at module level, or use functools.partial
# import functools
# def multiply(a, b):
# return a * b
# squarer = functools.partial(multiply, 2) # picklable
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
results = list(pool.map(double, [1, 2, 3, 4]))
print(results) # [2, 4, 6, 8]
When NOT to use multiprocessing·python
# Bad use cases — overhead dominates
#
# 1. Tiny tasks. The pickle + IPC + process startup overhead
# far exceeds the work itself.
#
# 2. I/O-bound work. asyncio or threading do this with much less overhead.
#
# 3. Anywhere you need shared state. Multiprocessing makes that hard.
# Use asyncio + a lock, or design without shared state.
#
# Good use cases:
# 1. Numeric work that can be parallelized — ML preprocessing,
# image processing, scientific simulations
# 2. Independent CPU-heavy tasks where each takes >>100ms
# 3. When you need true OS-level parallelism (different from threading)
# The bias today: try asyncio first for I/O. Use multiprocessing
# specifically for CPU-bound parallel work.
Define def fib(n) that recursively computes Fibonacci (deliberately slow — no caching). Use ProcessPoolExecutor to compute fib(35), fib(36), fib(37), fib(38) in parallel. Time it. Then time the sequential version. Compare. (You'll need if __name__ == '__main__': guard.) On a multi-core machine, the parallel version should be roughly 4x faster.
Progress
Progress is local-only — sign in to sync across devices.