The first rule of optimization: profile first. Without measurement, you'll spend hours speeding up code that wasn't slow to begin with. Python has stdlib tools for this: cProfile for "where is time being spent across this whole program," timeit for "how fast is this snippet."
cProfile — finding the hot spots
python -m cProfile -s cumulative script.py runs your script and prints function-call timing sorted by cumulative time. The top entries are where to focus optimization effort. cProfile has overhead but is built-in. py-spy (third-party) is sample-based and has near-zero overhead.
timeit — microbenchmarks
timeit.timeit("expression", number=1_000_000) times an expression repeatedly. timeit handles GC and best-of-N runs. Useful for "is this comprehension faster than that for-loop?" questions. Beware: microbenchmarks lie about real-world performance — they don't reflect cache behavior, real workloads, or amortized cost.
Memory profiling
tracemalloc is built-in: starts tracking allocations, takes a snapshot, shows top allocations by file/line. memray (Bloomberg) is a fancier external tool with a flame graph UI. For occasional "why is this using 8GB," tracemalloc is enough; for production memory profiling, memray.
Principle: Profile before optimizing. Optimize the actual bottleneck, not the imagined one. Then re-profile to confirm. The single most important habit in performance work — and the one most often skipped.
Code
cProfile — find the slow code·bash
# Run a script under cProfile, sort by cumulative time
python -m cProfile -s cumulative my_script.py
# Save to a file for later analysis
python -m cProfile -o profile.stats my_script.py
# Inspect saved profile
python -c "import pstats; p = pstats.Stats('profile.stats'); p.sort_stats('cumulative'); p.print_stats(20)"
# Or use snakeviz for a visual flame chart:
# pip install snakeviz
# snakeviz profile.stats
cProfile in code — for specific sections·python
import cProfile
import pstats
import io
def expensive():
return sum(i * i for i in range(1_000_000))
profiler = cProfile.Profile()
profiler.enable()
expensive()
profiler.disable()
# Print stats
s = io.StringIO()
ps = pstats.Stats(profiler, stream=s).sort_stats('cumulative')
ps.print_stats(20)
print(s.getvalue())
timeit — microbenchmarks·python
import timeit
# Compare two implementations
t1 = timeit.timeit(
'sum([x*x for x in range(1000)])',
number=10_000
)
t2 = timeit.timeit(
'sum(x*x for x in range(1000))',
number=10_000
)
print(f"list comp: {t1:.3f}s")
print(f"generator: {t2:.3f}s")
# Generator is slightly faster for big ranges (no list materialization)
# With setup code
t3 = timeit.timeit(
'd["key"]',
setup='d = {"key": 42}',
number=10_000_000,
)
print(f"dict access: {t3:.3f}s")
tracemalloc — memory profiling·python
import tracemalloc
tracemalloc.start()
# Allocate something
big = [list(range(1000)) for _ in range(1000)]
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:5]:
print(stat)
tracemalloc.stop()
# Useful for pinpointing where memory is going
# For deeper investigation, look at memray (Bloomberg's tool)
Use cProfile to profile a script that computes the 30th Fibonacci number recursively (without caching). Identify the most-called function. Then add @functools.cache and re-profile. Compare. Then use timeit to compare three approaches to summing the first 1000 squares: a for loop with +=, a list comprehension with sum(), and a generator expression with sum(). Print the times.
Progress
Progress is local-only — sign in to sync across devices.