Python Performance Optimization

Python Performance Optimization

Overview

Use this skill to investigate and improve Python performance with a measurement-first workflow.

The core operating model is:

1. define the user-visible performance problem,
2. collect a reproducible baseline,
3. choose the profiler that matches the symptom,
4. make one bounded change at a time,
5. re-measure under the same conditions.

This skill is for real diagnosis, not guesswork. Do not promise speedups from caching, vectorization, concurrency, or refactoring unless measurements on representative workloads show improvement.

The baseline workflow works with the Python standard library alone. Optional third-party profilers such as py-spy, pyperf, and Scalene can improve diagnosis when deterministic profiling is not enough, but they are not required.

Open these support files when needed:

• references/runtime-practices.md for profiler selection, benchmark hygiene, and memory/concurrency rules.
• examples/implementation-example.md for a concrete before/after optimization sequence.
• scripts/validate-runtime.py for repeat timing, cProfile summaries, and optional tracemalloc allocation diffs.

When to Use

Use this skill when the task involves one or more of these:

• Python code is slower than expected and the bottleneck is not yet known.
• A recent change increased latency, CPU time, or memory allocation.
• A user wants evidence before deciding whether to optimize algorithm, data structure, caching, I/O, or concurrency.
• Memory growth needs investigation with allocation tracing rather than assumptions.
• A proposed optimization needs before/after proof under the same interpreter, dependencies, and representative inputs.

Do not use this skill as the primary workflow when:

• the issue is mainly database tuning, network architecture, operating-system tuning, or infrastructure scaling,
• the operator cannot run representative workloads or gather measurements safely,
• the task asks for speculative “make it faster” advice without code, workload details, or acceptance criteria.

Operating Table

Workflow

1. Define the performance question.

   • Capture the metric that matters: request latency, batch runtime, throughput, peak allocation, or allocation growth over time.
   • Record the relevant workload, input size, interpreter version, dependency set, and platform assumptions.

2. Create a stable baseline before changing code.

   • Prefer repeated measurements over one-off timing.
   • Keep inputs, logging level, interpreter, and environment fixed.
   • If possible, test representative workloads instead of toy microbenchmarks.
   • Use scripts/validate-runtime.py or timeit/pyperf for repeatable timing.

3. Choose the profiler that matches the symptom.

   • Use cProfile first for Python-call hotspot analysis.
   • Use tracemalloc when the problem is allocation growth, unexpected memory pressure, or peak Python allocations.
   • Use py-spy optionally when you need low-overhead sampling of a live or hard-to-instrument process.
   • Use Scalene optionally when you need stronger attribution for Python time vs native time or mixed CPU/memory behavior.
   • See references/runtime-practices.md for the symptom-to-tool matrix.

4. Inspect the evidence carefully.

   • For cProfile, compare cumtime and tottime rather than scanning only call counts.
   • For memory, compare tracemalloc snapshots around the suspect operation.
   • Confirm whether the hotspot is algorithmic work, repeated conversion/parsing, I/O waiting, synchronization, serialization, or cache churn.

5. Apply one bounded optimization at a time. Examples:

   • replace repeated O(n) lookups with a dict or set,
   • reduce unnecessary allocations or copies,
   • move repeated pure-function work behind a bounded cache,
   • batch I/O instead of performing many tiny operations,
   • consider process-based parallelism for CPU-bound work only after measuring task granularity and overhead.

6. Re-measure under the same conditions.

   • Use the same dataset, interpreter, and environment.
   • Report baseline vs changed results with units and repetition count.
   • Note tradeoffs such as higher memory use, reduced readability, or changed concurrency behavior.

7. Stop if evidence does not support the change.

   • Revert speculative optimizations.
   • Escalate to a different tool or a broader architecture review when the current measurement method cannot explain the bottleneck.

Troubleshooting