Profile with cProfile & timeit

You have a Python script that takes 30 seconds to run, and your manager asks you to make it faster. You stare at the code, unsure where to start — guessing at bottlenecks wastes hours. Python includes two built-in profilers — cProfile and timeit — that pinpoint exactly what's slow and by how much. This lesson shows how to profile code with cProfile and timeit so you eliminate performance guesswork.

The problem this lesson solves

Performance tuning without profiling is like building a house in the dark — you might move a wall that was never the issue. Poorly optimized Python code leads to late batch jobs, sluggish APIs, and cloud bills that keep growing.

Common scenarios include: - A for loop that runs a thousand unnecessary database queries. - A function that recomputes the same result every call instead of caching. - An innocent-looking str += in a loop that builds a 100 MB string.

Profiling reveals the actual hotspots — not the ones you think are slow. With cProfile you get a function‑by‑function breakdown of call count and cumulative time. With timeit you measure small code snippets with microsecond precision.

Core concept / mental model

Think of profiling as a performance microscope.

Mental model: cProfile is a heat map of your whole city — it shows the busiest intersections. timeit is a stopwatch for a single street corner — it tells you exactly how long one block takes.

How it works step by step

1. Profile an entire script with cProfile

The easiest way is from the command line:

python -m cProfile my_script.py

This prints a table to stdout with columns: - ncalls — number of calls - tottime — total time spent in the function itself (excluding subcalls) - cumtime — total time including all subcalls - percall — time per call - filename:lineno(function) — the function identity

2. Use pstats to sort and filter programmatically

import pstats
from cProfile import Profile

profiler = Profile()
profiler.enable()
# run your code here …
profiler.disable()

stats = pstats.Stats(profiler).sort_stats('cumtime')
stats.print_stats(10)   # top 10 slowest

3. Time a snippet with timeit

The module provides a Timer class:

import timeit

t = timeit.Timer(stmt="''.join(str(n) for n in range(1000))")
print(t.timeit(number=100000))

Or from the command line:

python -m timeit -s "import mymodule" "mymodule.heavy_function()"

The -n flag sets the number of loops (auto‑guessed if omitted).

Hands-on walkthrough

Example 1: cProfile on a slow script

Suppose we have slow.py:

import time

def compute():
    total = 0
    for i in range(1_000_000):
        total += i ** 2
    return total

def main():
    compute()
    time.sleep(0.1)

if __name__ == '__main__':
    main()

Run profiling:

python -m cProfile slow.py

Expected output (top portion):

         6 function calls in 0.286 seconds

   Ordered by: standard name

   ncalls  tottime  cumtime  filename:lineno(function)
        1    0.186    0.186   slow.py:3(compute)
        1    0.000    0.286   slow.py:8(main)
        1    0.000    0.286   <string>:1(<module>)
        1    0.100    0.100   {built-in method time.sleep}
        1    0.000    0.000   {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000   {method 'append' of 'list' objects}

Pro tip: The sleep() call masquerades as cumtime inside main. Always check tottime for the actual CPU cost of your Python code.

Example 2: Compare list building with timeit

import timeit

setup = "import random; data = [random.randint(1,100) for _ in range(1000)]"

stmt_a = "result = []\nfor x in data:\n    result.append(x * 2)"
stmt_b = "result = [x * 2 for x in data]"


t_a = timeit.timeit(stmt_a, setup, number=100000)
t_b = timeit.timeit(stmt_b, setup, number=100000)

print(f"For loop : {t_a:.5f}s")
print(f"Comprehension : {t_b:.5f}s")

Expected output (approximate):

For loop : 0.34501s
Comprehension : 0.26102s

Example 3: Profile a single function with cProfile + pstats

import cProfile, pstats

def heavy_io():
    with open('output.txt', 'w') as f:
        for i in range(100_000):
            f.write(f"Line {i}\n")

def main():
    profiler = cProfile.Profile()
    profiler.enable()
    heavy_io()
    profiler.disable()

    stats = pstats.Stats(profiler)
    stats.sort_stats('cumtime')
    stats.print_stats(10)

if __name__ == '__main__':
    main()

Expected output shows heavy_io and write as the top entries.

Compare options / when to choose what

Pro tip: Start with python -m cProfile -s cumtime my_script.py to see the total time spent in each function (including its children). Then switch to timeit for head‑to‑head comparisons.

Troubleshooting & edge cases

1. cProfile hides import time

If your script imports many heavy modules, the first run includes import overhead. To measure only runtime performance, import all modules before enabling the profiler.

2. timeit returns wildly different numbers each run

Your system may be under load. Increase number to at least 10 000 and ignore the first run (warm‑up). Use timeit.repeat(…) and take the minimum.

import timeit
results = timeit.repeat(
    "sum(range(1000))",
    repeat=5,
    number=10000
)
print(min(results))  # discard outliers

3. Profiling multithreaded code

cProfile only tracks the thread where it's enabled. Use threading.Thread(target=run_profiled, …) and enable the profiler inside the thread.

4. Infinite loops or long sleeps

If your script never finishes, press Ctrl+C — cProfile loses data. Use profiler.runcall(func) with a timeout wrapper for safe profiling.

What you learned & what's next

You now have two essential tools for Python performance work: - cProfile — full‑program profiling, call counts, and cumulative time. - timeit — precise, isolated timing for comparisons.

Wrapping everything together:

import cProfile, pstats, timeit

def fast():
    return sum([x*2 for x in range(1000)])

def slow():
    total = 0
    for x in range(1000):
        total += x * 2
    return total

# Compare microbenchmarks
print("Fast:", timeit.timeit(fast, number=10000))
print("Slow:", timeit.timeit(slow, number=10000))

# Profile the whole script
profiler = cProfile.Profile()
profiler.enable()
fast()
slow()
profiler.disable()
pstats.Stats(profiler).sort_stats('tottime').print_stats(10)

Next up: You'll learn memory profiling with tracemalloc — because speed isn't everything; memory leaks can hurt just as much.