Maintaining Python Tests

Maintaining Python tests

Before you propose a change to how the suite runs in CI, check things already tried. It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.

Use this skill for an existing Python test suite. Use /writing-tests before adding or substantially changing coverage. Use /fixing-flaky-tests when intermittent failure is the main problem.

The goal is not a smaller test count. The goal is a suite that catches the same realistic regressions with less compute, less waiting, and less maintenance.

Principles

1. Measure before changing code. Rank tests by total observed work, not by one slow local run.
2. Preserve behavior coverage. Keep cases that exercise different validation, persistence, integration, or output paths.
3. Remove only expired or redundant coverage. Get explicit approval before deleting a test.
4. Share expensive infrastructure, not mutable test state. Preserve isolation with unique IDs, schemas, tables, topics, or tenants.
5. Measure after merge. Local results prove the mechanism. Fresh master data proves the result in CI.
6. Separate testcase work from suite wall time. A change can reduce summed testcase time and not change the slowest pytest suite.

Read measurement.md before you query timing data or report an improvement. Read optimization-patterns.md when you select a fix.

Workflow

1. Define the result

Write down the user problem before choosing a test:

• Reduce total test compute.
• Reduce the slowest pytest suite.
• Remove expired maintenance burden.
• Restore test ownership.
• Reduce repeated external-service setup.

These results need different measurements. Do not claim faster CI when only summed test call time decreased.

2. Rank current work

Use recent PostHog test spans from master when available. Start with a complete window after the latest relevant merge.

Rank at least three views:

• Individual tests: execution count multiplied by duration.
• Parameterized families: all cases under one test function or fixture family.
• Pytest suites or shards: root-span wall time and the slowest suite per run attempt.

Use p50 to find steady cost. Use p95 to find contention or tail behavior. Use sampled observed hours to find repeated medium-cost tests. Use root-span testcase totals for complete pytest work.

Do not select a target from an old ranking after several fixes merge. Rebuild the ranking first.

Do not rank from .test_durations. It holds flat default values (0.01, 18.0, and 60.0) for tests that pytest-split could not time. These values are not measurements.

3. State why the test exists

Apply the /writing-tests gate to every target:

What realistic regression does this test catch that no existing test already catches?

Then classify each case:

• Distinct behavior: keep it.
• Same behavior with representative inputs: parameterize it.
• Framework or implementation detail: replace it with an observable assertion, or propose to delete it.
• Temporary migration coverage: check whether every supported environment completed the migration.
• Runnable backfill or reusable migration system: keep active behavior coverage.

Do not infer redundancy from similar names. Read the setup, execution path, assertions, and production entry point.

4. Establish a baseline

Run the exact target with the same command that you will use after the change. Record:

• Test call time.
• Full command wall time.
• Setup and teardown time when available.
• Number of collected and executed cases.
• Whether the run was cold or warm.

Run the surrounding class or file when fixtures can change the result. A single test can hide repeated setup that only appears across the family.

Do not compare full pytest wall time with summed call time. They measure different work.

5. Find the cost center

Profile before rewriting. Attribute time to one of these groups:

• Test collection or environment boot.
• Fixture setup or teardown.
• Database creation, flush, or migration.
• Worker, consumer, broker, or container startup.
• Product code executed by the test.
• Snapshot serialization or formatting.
• Polling, retries, or real time.

Use a wall-clock profile for subprocesses and I/O. Use a CPU profile for Python or JavaScript work. A CPU profile can miss time spent in services or child processes.

6. Select the smallest safe fix

Prefer these options in order:

1. Remove expired coverage with explicit approval.
2. Replace incidental implementation assertions with stronger observable assertions.
3. Move the test to a cheaper level.
4. Reuse expensive immutable infrastructure across cases.
5. Reuse production objects only when the production path repeats unnecessary work.
6. Remove a distinct case only when another test proves the same regression through the same boundary.

Do not add timing assertions. CI timing is too noisy for a correctness test.