Fixing Flaky Tests

Fixing flaky tests

Before you propose a change to how the suite runs in CI, check things already tried. It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.

Three non-negotiables, in order:

1. Reproduce before you fix. A fix for a failure you never observed is a guess. Only fall back to analytical fixes when the escalation ladder below is exhausted.
2. Fix the root cause. Sleeps, raised timeouts, retries, and weakened assertions hide flakes; they do not fix them.
3. Validate with an N-run loop. One green run proves nothing about an intermittent failure. Size N to the observed failure rate.

Stabilizing the test is not the only valid ending. Once you know why it flakes, step 5 asks whether it should exist. A test that catches nothing real is worth deleting, and one that flakes because of the level it runs at is worth moving down a rung.

Before any of these: measure, don't assume. Flaky-vs-deterministic, and the rate, are facts to establish from verifiable GitHub run data (step 1) — never inherited from a Slack alert, a teammate's guess, or a ci:insights label.

For triaging a red CI run (finding and classifying the failure), use the debugging-ci-failures skill first — this skill takes over once the failure is classified as a flaky test. For writing new Playwright tests that aren't flaky, use the playwright-test skill. investigating-ci-failures (green/red boundary) and diagnosing-ci-and-merge-bottlenecks (the engineering-analytics-flaky-tests tool's caveats) are product skills under products/engineering_analytics/skills/, not invocable here: read their SKILL.md at that path.

1. Measure the failure rate — from GitHub, not from a digest

The GitHub Actions API (or GitHub MCP) is the source of truth. hogli ci:insights is a digest, not an oracle — it can mislabel flaky-vs-deterministic, it lags the API until GitHub's webhook settles, and it cannot give you a rate at all: it reports absolute counts, because CI emits failures but omits ordinary passing runs, so there is no denominator. Use it to validate a hypothesis or pull historical context, never as the first move or the classification authority.

Establish the timeline yourself from raw run data:

# Pass/fail history for the workflow on the branch under suspicion (master shown):
gh run list --repo PostHog/posthog --workflow=ci-backend.yml --branch master \
  --status completed --limit 60 --json conclusion,headSha,createdAt,databaseId

The run-level conclusion is not enough — a red run may be failing on a different job/test. Confirm it is the same test each time by reading the failing shard's log; and for green runs, confirm the test actually ran and passed in that shard (it may have been sharded elsewhere, not fixed):

gh api repos/PostHog/posthog/actions/jobs/{job_id}/logs \
  | grep -E "<exact::test::id>|short test summary"

Read the timeline before you classify:

• Interleaved pass/fail on adjacent commits — the same unchanged test verified passing in some runs, failing in others → genuinely flaky; continue here.
• A long unbroken failure streak (say 30+ consecutive) is statistically incompatible with a flake — at any per-run rate below ~95%, p^30 ≈ 0. That is a deterministic regression: go to debugging-ci-failures and find the introducing commit (step 4).
• Every rerun fails on one runner, yet master passes the same code → an environment-dependent race, not a regression: kernel clock granularity, filesystem, runner speed. Retrying on the same runner cannot help, so "failed 3/3 attempts" says nothing about whether the PR caused it. A merge-queue failure hands you the control for free: the trunk-merge/** test PR's body names the master SHA it is based on, so read master's run at that SHA (gh api repos/PostHog/posthog/commits/<sha>/check-runs) and confirm the same test ran and passed there.
• Both at once is common: a latent flake whose rate jumped to ~100%. Find the transition (last green → first red); that boundary, not the digest's one-line verdict, is what tells you what tipped it. products/engineering_analytics/skills/investigating-ci-failures/ has the boundary query if the failure reached master.

If a failure is reported as (or you suspect it is) consistent, don't serialize — measure the rate and attempt a repro in parallel.

Record the measured rate (failures / total runs, from the run data). You need it to size the validation loop in step 7.

Then confirm it is not already handled:

git log --oneline -10 -- <test_file_path>          # recently fixed already?
hogli ci:insights search "<test name or error>"    # cross-run history — corroborate against the run data, do not trust blindly

search reports two surfaces; read each for what it actually claims:

• broken tests (recent failure fingerprints, last 2 days). A potentially_resolved state means that job's latest default-branch run is green again — weak evidence a fix landed, not proof. Confirm against the run data that it covers this failure before reporting instead of re-fixing.
• test health (ranked by blast radius; the same spans the engineering-analytics-flaky-tests MCP tool reads). confirmed_flake is the only classification backed by proof: one commit both failed and passed the test in the same matrix job, via a re-run attempt going green or an in-job retry. A pass in a different matrix leg is not recovery. suspected_regression means no recovery was recorded — absence of proof, not proof of a regression, so treat it as real until your own run data says otherwise.

Corroborate with Trunk Flaky Tests

CI uploads test results to Trunk Flaky Tests, which tracks per-test failure history across master and PR runs. On a PR, the trunk-io bot's "Trunk Test Analytics" comment links that PR's slice (https://app.trunk.io/posthog-inc/flaky-tests/pr/<number>?repo=PostHog/posthog).

The trunk MCP server in .mcp.json queries it (tools are marked experimental by Trunk):

• search-test (repoName: "PostHog/posthog", testNameSearch: "<test name, no filepath>") → the test case ID.
• fix-flaky-test (repoName, testCaseId) → failure history, first-seen commit, git blame, and Trunk's root-cause investigation; createNewInvestigation: true triggers a fresh analysis (takes up to a minute).

Authenticate once via /mcp → trunk (browser OAuth); headless environments instead add an Authorization: Bearer header with a TRUNK_API_TOKEN org token to the server entry.

Two limits worth knowing before you start here. AI investigations are not enabled for this repo, so fix-flaky-test returns history or nothing, never a root cause. And lookup only goes name to ID: a bare dashboard link identifies a test you cannot name, so ask for the test name rather than guessing at the ID.

Trunk attributes each test to a team through CODEOWNERS, which cannot express the owners.yaml map. .github/scripts/trunk-codeowners.sh projects the map into a generated CODEOWNERS before each upload (hogli owners:codeowners builds the same file locally), so a test's owner in Trunk should match hogli owners:who. Where it does not, the projection dropped a spelling two teams would both claim.

Like ci:insights, this is corroboration and history, not the classification authority — flaky-vs-deterministic and the rate still come from the run data above.

2. Extract the failure from CI

gh run view <run-id> --log-failed
# If the job was re-run and the latest attempt passed, the flaky failure
# lives in a previous attempt:
gh api "repos/PostHog/posthog/actions/runs/{run_id}/attempts/1/jobs"
gh api "repos/PostHog/posthog/actions/jobs/{job_id}/logs"

Capture before moving on:

• Exact test ID (file path + test name) and the failing assertion or error.
• Surrounding warnings — [MSW] Unhandled, async leak warnings, teardown errors — these are often the actual cause, printed before the symptom.
• Which other tests ran in the same worker/shard before it (ordering suspects).

3. Reproduce locally — before touching anything

Escalate through these conditions until the failure appears. Stop at the first level that reproduces it; that level is your validation environment for step 7.

1. Single run: hogli test <path>::<test> — confirms the test runs at all.

2. Repetition loop (default N=20): catches probabilistic flakes.

3. CI-like conditions: CI runs Jest sharded with low worker counts on contended runners — read the current flags from frontend/package.json's test script and .github/workflows/ci-frontend.yml before running. Example (with the flags as of this writing), running the test alongside its shard neighbors:

   pnpm --filter=@posthog/frontend jest <test_file> <neighbor_file> --maxWorkers=2 --forceExit

   For pytest, run the whole file or class rather than the single test, so module-level fixtures and ordering match CI.

4. Ordering: run suspected polluter tests before the victim; reverse the order within the file. Flakes that vanish in isolation are ordering bugs.

5. Contention: re-run the loop while something CPU-heavy runs in another shell (e.g. a parallel full-file jest run). Timeout-class flakes often only show here.

The loop harness — judge by exit code, not by grepping output:

N=20; PASS=0; FAIL=0
for i in $(seq 1 $N); do
  if <test command> >/tmp/flake-run.log 2>&1; then
    PASS=$((PASS+1))
  else
    FAIL=$((FAIL+1)); cp /tmp/flake-run.log /tmp/flake-fail-$i.log; echo "run $i: FAIL"
  fi
done
echo "$PASS passed, $FAIL failed out of $N"

Two cost notes for the loop:

• While reproducing, break after the first failure — one captured failure log is enough. Complete all N runs only when measuring the failure rate or validating in step 7.
• The pnpm --filter=@posthog/frontend jest script runs pnpm build:products before every invocation. Inside a loop, build once, then iterate with pnpm --filter=@posthog/frontend exec jest ..., which skips the rebuild.

If nothing reproduces after the full ladder, the flake is CI-environment-specific. Before settling for that, ask what the runner has that your machine lacks: Linux mtimes from a coarse clock where macOS gives nanoseconds, a case-sensitive filesystem, a different timezone, /tmp on a different filesystem. Often you can force the CI condition instead of waiting for it — os.utime the files into the ordering the coarse clock produced, run under TZ=UTC — and then validate empirically after all. Otherwise proceed with a fix grounded in the CI evidence and root-cause analysis, and say so explicitly in the report — the validation in step 7 is then analytical, not empirical.

4. Root-cause the flake

Match the symptom to a cause class; never patch the symptom.

When the cause isn't obvious, bisect

If the symptom table doesn't point at a clear cause and the test file itself is unchanged (git log -- <test_file> is stale), the trigger is elsewhere — a neighbor test, a dependency bump, or a product change. Find when it started instead of guessing:

• Bisect the CI run history first (cheap, no local builds): from the step-1 timeline, take the last-green → first-red boundary and diff the commits in that window (git log <good>..<bad>). That short list often names the culprit outright.

• git bisect the code when you can reproduce locally and the failure is (near-)deterministic:

  git bisect start <bad-sha> <good-sha>
  git bisect run bash -c '<repro command>'   # exit 0 = good, non-zero = bad

  Caveat: for an intermittent flake, a lucky pass at a step sends git bisect down the wrong path. Trust code-bisect only when the failure is deterministic; otherwise run the repro N times per step (fail if any iteration fails), or just use the CI run-history boundary.

PostHog-specific patterns: