Profiling and Optimization

Profiling is the process of analyzing your code to identify performance bottlenecks. In Python, you can measure execution time of small snippets using the timeit module, and perform full program profiling using the cProfile module.

Micro-benchmarking with timeit

timeit exists because naive timing lies. It runs your snippet many times, uses the best available clock, and reports the total — sidestepping the noise that makes a single time.time() measurement meaningless.

Basic timeit execution

By default, timeit() executes the snippet 1,000,000 times. Here we tell it to execute only 10 times.

import timeit

# Measure execution time
t = timeit.timeit('"-".join(str(n) for n in range(100))', number=10)
print(isinstance(t, float))

True

Disabling Garbage Collection

It also disables garbage collection while timing, so an unrelated collection cycle cannot land in the middle of your measurement and make one run look mysteriously slow.

import timeit

# Even an empty 'pass' statement can be timed
t = timeit.timeit('pass', number=10)
print(isinstance(t, float))

True

The Default Timer

Timing uses time.perf_counter(), the highest-resolution clock available, and reports float seconds. It measures elapsed wall-clock time, which is what you actually care about.

import timeit

t = timeit.default_timer()
print(isinstance(t, float))

True

Using timeit.repeat()

timeit.repeat() acts just like timeit(), but it runs the whole process multiple times and returns a list of results. You can use the lowest time to find the best possible performance without OS interference.

import timeit

t = timeit.Timer('1+1')
res = t.repeat(repeat=3, number=10)
print(len(res) == 3)

True

Deterministic Profiling with cProfile

For whole programs, reach for cProfile. It is a C extension with low enough overhead to profile real workloads, unlike the pure-Python profile module it replaces.

cProfile tells you exactly how many times each function was called, and how much time was spent inside it.

Basic cProfile execution

import cProfile
import io
import pstats

pr = cProfile.Profile()
pr.enable()

def my_func():
    pass

my_func()

pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s)
ps.print_stats()
print('function calls' in s.getvalue())

True

Analyzing Output with pstats

Raw profile data is not meant to be read directly. pstats sorts and formats it — sort_stats('cumtime') to find the expensive call tree, 'tottime' to find the hot function itself.

You can sort your profiler statistics to quickly identify bottlenecks, such as sorting by cumulative time.

import cProfile
import pstats

profiler = cProfile.Profile()
profiler.enable()

# Perform some work
sorted([3, 1, 2])

profiler.disable()

# Load stats into pstats and sort by cumulative time
stats = pstats.Stats(profiler)
stats = stats.sort_stats('cumtime')

print(isinstance(stats, pstats.Stats))

True

Interactive Iteration Visualization

Use the interactive slider below to change the number of loop iterations, and time how long a simple mathematical operation takes as the number of iterations grows!

import timeit

iterations = 1000
t = timeit.timeit('sum(i*i for i in range(100))', number=iterations)
print(f"Time for {iterations} iterations: {t:.4f} seconds")

Time for 1000 iterations: 0.0039 seconds