Multiprocessing — True Parallelism for CPU-Bound Work
~18 min · multiprocessing, process, parallel, executor
Level 0Curious
0 XP0/93 lessons0/23 achievements
0/100 XP to next level100 XP to go0% complete
Why multiprocessing exists
The GIL prevents two threads from running Python bytecode in parallel. Multiprocessing sidesteps this by using separate Python processes — each with its own GIL, its own memory, its own everything. For CPU-bound work that you want to actually parallelize, this is the answer.
The cost — process startup and serialization
Starting a process is far slower than starting a thread (milliseconds vs microseconds). Passing data between processes requires serialization — Python objects pickle/unpickle across the boundary. So multiprocessing wins when the work-per-task is bigger than the startup + communication overhead. For a 1-second computation, the overhead is negligible. For a 1ms task, threading or no concurrency is faster.
ProcessPoolExecutor — the easy way
concurrent.futures.ProcessPoolExecutor has the same API as ThreadPoolExecutor. Just swap the class. The function and arguments must be picklable, which means: defined at the top level (not inside another function), the arguments must be standard types (not lambdas, not file handles, etc.). This catches people the first time.
Sharing state — Manager and shared memory
Processes don't share memory by default. multiprocessing.Manager gives you proxies to shared lists/dicts that work across processes (with overhead). multiprocessing.shared_memory (3.8+) gives raw shared memory blocks for high-throughput numeric data. Most multiprocessing code doesn't need these — pass arguments in, return results out, no shared state.
Principle: Multiprocessing for CPU-bound. Asyncio (or threading) for I/O-bound. The wrong choice is the most common production mistake — like running an HTTP server with multiprocessing or a CPU-heavy computation in asyncio.
Code
ProcessPoolExecutor — same API, different model·python
from concurrent.futures import ProcessPoolExecutor
import time
def compute_squares(n):
return sum(i * i for i in range(n))
if __name__ == "__main__": # required for multiprocessing on some platforms
start = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as pool:
results = list(pool.map(compute_squares, [1_000_000] * 4))
print(f"results count: {len(results)}")
print(f"elapsed: {time.perf_counter() - start:.2f}s")
# With 4 processes on a multi-core machine, runs ~4x faster
# than the same in a single process or with ThreadPoolExecutor
# (because of the GIL)
if __name__ == '__main__': — why it matters·python
# On Windows and macOS-spawn, multiprocessing imports the script
# in each worker. Without the if __name__ guard, child workers
# would re-execute the multiprocessing setup and recursively spawn.
#
# Always guard top-level multiprocessing code:
#
# def work(x):
# return x * 2
#
# if __name__ == '__main__':
# with ProcessPoolExecutor() as pool:
# results = list(pool.map(work, range(10)))
# print(results)
# Functions must be picklable — defined at module top level
Pickling — what works, what doesn't·python
from concurrent.futures import ProcessPoolExecutor
# This works — module-level function, picklable args
def double(x):
return x * 2
# This DOESN'T work — lambda is not picklable
# squarer = lambda x: x * x
# pool.submit(squarer, 5) # PicklingError
# Workaround — define the function at module level, or use functools.partial
# import functools
# def multiply(a, b):
# return a * b
# squarer = functools.partial(multiply, 2) # picklable
if __name__ == '__main__':
with ProcessPoolExecutor() as pool:
results = list(pool.map(double, [1, 2, 3, 4]))
print(results) # [2, 4, 6, 8]
When NOT to use multiprocessing·python
# Bad use cases — overhead dominates
#
# 1. Tiny tasks. The pickle + IPC + process startup overhead
# far exceeds the work itself.
#
# 2. I/O-bound work. asyncio or threading do this with much less overhead.
#
# 3. Anywhere you need shared state. Multiprocessing makes that hard.
# Use asyncio + a lock, or design without shared state.
#
# Good use cases:
# 1. Numeric work that can be parallelized — ML preprocessing,
# image processing, scientific simulations
# 2. Independent CPU-heavy tasks where each takes >>100ms
# 3. When you need true OS-level parallelism (different from threading)
# The bias today: try asyncio first for I/O. Use multiprocessing
# specifically for CPU-bound parallel work.
Define def fib(n) that recursively computes Fibonacci (deliberately slow — no caching). Use ProcessPoolExecutor to compute fib(35), fib(36), fib(37), fib(38) in parallel. Time it. Then time the sequential version. Compare. (You'll need if __name__ == '__main__': guard.) On a multi-core machine, the parallel version should be roughly 4x faster.
Progress
Progress is local-only — sign in to sync across devices.