Profiling and Optimization

Profiling is the process of analyzing your code to identify performance bottlenecks. In Python, you can measure execution time of small snippets using the timeit module, and perform full program profiling using the cProfile module.

Micro-benchmarking with timeit

timeit exists because naive timing lies. It runs your snippet many times, uses the best available clock, and reports the total — sidestepping the noise that makes a single time.time() measurement meaningless.

Basic timeit execution

By default, timeit() executes the snippet 1,000,000 times. Here we tell it to execute only 10 times.

import timeit

# Measure execution time
t = timeit.timeit('"-".join(str(n) for n in range(100))', number=10)
print(isinstance(t, float))

True

Disabling Garbage Collection

It also disables garbage collection while timing, so an unrelated collection cycle cannot land in the middle of your measurement and make one run look mysteriously slow.

import timeit

# Even an empty 'pass' statement can be timed
t = timeit.timeit('pass', number=10)
print(isinstance(t, float))

True

The Default Timer

Timing uses time.perf_counter(), the highest-resolution clock available, and reports float seconds. It measures elapsed wall-clock time, which is what you actually care about.

import timeit

t = timeit.default_timer()
print(isinstance(t, float))

True

Using timeit.repeat()

timeit.repeat() acts just like timeit(), but it runs the whole process multiple times and returns a list of results. You can use the lowest time to find the best possible performance without OS interference.

import timeit

t = timeit.Timer('1+1')
res = t.repeat(repeat=3, number=10)
print(len(res) == 3)

True

Deterministic Profiling with cProfile

For whole programs, reach for cProfile. It is a C extension with low enough overhead to profile real workloads, unlike the pure-Python profile module it replaces.

cProfile tells you exactly how many times each function was called, and how much time was spent inside it.

Basic cProfile execution

import cProfile
import io
import pstats

pr = cProfile.Profile()
pr.enable()

def my_func():
    pass

my_func()

pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s)
ps.print_stats()
print('function calls' in s.getvalue())

True

Analyzing Output with pstats

Raw profile data is not meant to be read directly. pstats sorts and formats it — sort_stats('cumtime') to find the expensive call tree, 'tottime' to find the hot function itself.

You can sort your profiler statistics to quickly identify bottlenecks, such as sorting by cumulative time.

import cProfile
import pstats

profiler = cProfile.Profile()
profiler.enable()

# Perform some work
sorted([3, 1, 2])

profiler.disable()

# Load stats into pstats and sort by cumulative time
stats = pstats.Stats(profiler)
stats = stats.sort_stats('cumtime')

print(isinstance(stats, pstats.Stats))

True

Interactive Iteration Visualization

Drag the slider to change the number of iterations and watch the measured time grow with it. The exact figure will differ on every run and on every machine — that variability is the reason timeit exists, and the reason no fixed output is printed here.

import timeit

iterations = 1000
t = timeit.timeit('sum(i*i for i in range(100))', number=iterations)
print(f"Time for {iterations} iterations: {t:.4f} seconds")