Triage CI Flake

Triage CI Failure

Overview

Systematic workflow designed specifically for triaging and fixing end-to-end (e2e) test failures in CI, especially flaky Playwright tests that pass locally but fail in CI. E2E tests that made it to main are usually flaky due to timing, bundling, or environment differences.

CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.

When to Use

• CI test fails on main branch after PR was merged
• Test passes locally but fails in CI
• Test failure labeled as "flaky" or intermittent
• E2E test timing out in CI only
• User provides a PR URL/number with failing checks

PR-Based Workflow (When PR URL/Number Provided)

When the user provides a PR URL (e.g., https://github.com/payloadcms/payload/pull/16701) or PR number:

Step 1: Fetch PR Status Checks

First, use tool_search with query "github pull request status checks" to load the GitHub tools.

Then use the github-pull-request_pullRequestStatusChecks tool to get all failing checks:

Tool: github-pull-request_pullRequestStatusChecks
Parameters:
  pullRequestNumber: <extracted from URL or provided>
  repo: { owner: "payloadcms", name: "payload" }

Step 2: Aggregate Failing Tests

Parse the status checks response and create a summary table:

For each failing check, the context field contains the test suite name (e.g., admin__e2e__list-view).

Step 3: Extract Failure Details from Each Check

For each failing check:

1. Visit the targetUrl to get detailed failure logs
2. Extract the specific test name and error message
3. Add to the aggregated failure list

Present a consolidated summary:

## Failing Tests Summary

### 1. admin**e2e**list-view (3/4)

- **Test**: "should use custom pagination limit"
- **Error**: Locator `.per-page__button` not found
- **File**: test/admin/e2e/list-view/e2e.spec.ts:1495

### 2. plugin-import-export

- **Test**: "should inherit limit from list view URL"
- **Error**: Locator `.per-page__button` not found
- **File**: test/plugin-import-export/e2e.spec.ts:150

Step 4: Identify Common Patterns

Look for patterns across failures:

• Same selector errors → likely a component change needing test updates
• Same test file → localized issue
• Different errors in same suite → may be test pollution

Step 5: Proceed with Triage

For each unique failure, follow the standard reproduction workflow below.

MANDATORY First Steps

YOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.

1. Extract suite name, test name, and error from CI logs
2. EXECUTE: Kill port 3000 to avoid conflicts
3. EXECUTE: pnpm dev $SUITE_NAME (use run_in_background=true)
4. EXECUTE: Wait for server to be ready (check with curl or sleep)
5. EXECUTE: Run the specific failing test with Playwright directly (npx playwright test test/TEST_SUITE_NAME/e2e.spec.ts:31:3 --headed -g "TEST_DESCRIPTION_TARGET_GOES_HERE")
6. If test passes, EXECUTE: pnpm prepare-run-test-against-prod
7. EXECUTE: pnpm dev:prod $SUITE_NAME and run test again

Only after EXECUTING these commands and seeing their output can you proceed to analysis and fixes.

"Analysis from logs" is NOT reproduction. You must RUN the commands.

Core Workflow

digraph triage_ci {
    "CI failure reported" [shape=box];
    "Extract details from CI logs" [shape=box];
    "Identify suite and test name" [shape=box];
    "Run dev server: pnpm dev $SUITE" [shape=box];
    "Run specific test by name" [shape=box];
    "Did test fail?" [shape=diamond];
    "Debug with dev code" [shape=box];
    "Run prepare-run-test-against-prod" [shape=box];
    "Run: pnpm dev:prod $SUITE" [shape=box];
    "Run specific test again" [shape=box];
    "Did test fail now?" [shape=diamond];
    "Debug bundling issue" [shape=box];
    "Unable to reproduce - check logs" [shape=box];
    "Fix and verify" [shape=box];

    "CI failure reported" -> "Extract details from CI logs";
    "Extract details from CI logs" -> "Identify suite and test name";
    "Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
    "Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
    "Run specific test by name" -> "Did test fail?";
    "Did test fail?" -> "Debug with dev code" [label="yes"];
    "Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
    "Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
    "Run: pnpm dev:prod $SUITE" -> "Run specific test again";
    "Run specific test again" -> "Did test fail now?";
    "Did test fail now?" -> "Debug bundling issue" [label="yes"];
    "Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
    "Debug with dev code" -> "Fix and verify";
    "Debug bundling issue" -> "Fix and verify";
}

Step-by-Step Process

1. Extract CI Details

From CI logs or GitHub Actions URL, identify:

• Suite name: Directory name (e.g., i18n, fields, lexical)
• Test file: Full path (e.g., test/i18n/e2e.spec.ts)
• Test name: Exact test description
• Error message: Full stack trace
• Test type: E2E (Playwright)

2. Reproduce with Dev Code

CRITICAL: Always run the specific test by name, not the full suite.

SERVER MANAGEMENT RULES:

1. ALWAYS kill all servers before starting a new one
2. NEVER assume ports are free
3. ALWAYS wait for server ready confirmation before running tests

# ========================================
# STEP 2A: STOP ALL SERVERS
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"

# ========================================
# STEP 2B: START DEV SERVER
# ========================================
# Start dev server with the suite (in background with run_in_background=true)
pnpm dev $SUITE_NAME

# ========================================
# STEP 2C: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"

# ========================================
# STEP 2D: RUN SPECIFIC TEST
# ========================================
# Run ONLY the specific failing test using Playwright directly
# For E2E tests (DO NOT use pnpm test:e2e as it spawns its own server):
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"

Did the test fail?

• ✅ YES: You reproduced it! Proceed to debug with dev code.
• ❌ NO: Continue to step 3 (bundled code test).

3. Reproduce with Bundled Code

If test passed with dev code, the issue is likely in bundled/production code.

IMPORTANT: You MUST stop the dev server before starting prod server.

# ========================================
# STEP 3A: STOP ALL SERVERS (INCLUDING DEV SERVER FROM STEP 2)
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"

# ========================================
# STEP 3B: BUILD AND PACK FOR PROD
# ========================================
# Build all packages and pack them (this takes time - be patient)
pnpm prepare-run-test-against-prod

# ========================================
# STEP 3C: START PROD SERVER
# ========================================
# Start prod dev server (in background with run_in_background=true)
pnpm dev:prod $SUITE_NAME

# ========================================
# STEP 3D: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"

# ========================================
# STEP 3E: RUN SPECIFIC TEST
# ========================================
# Run the specific test again using Playwright directly
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"

Did the test fail now?

• ✅ YES: Bundling or production build issue. Look for:
  • Missing exports in package.json
  • Build configuration problems
  • Code that behaves differently when bundled
• ❌ NO: Unable to reproduce locally. Proceed to step 4.

4. Unable to Reproduce

If you cannot reproduce locally after both attempts:

• Review CI logs more carefully for environment differences
• Check for race conditions (run test multiple times: for i in {1..10}; do pnpm test:e2e...; done)
• Look for CI-specific constraints (memory, CPU, timing)
• Consider if it's a true race condition that's highly timing-dependent

Common Flaky Test Patterns

Race Conditions

• Page navigating while assertions run
• Network requests not settled before assertions
• State updates not completed

Fix patterns:

• Use Playwright's web-first assertions (toBeVisible(), toHaveText())
• Wait for specific conditions, not arbitrary timeouts
• Use waitForFunction() with condition checks

Test Pollution

• Tests leaving data in database
• Shared state between tests
• Missing cleanup in afterEach

Fix patterns:

• Track created IDs and clean up in afterEach
• Use isolated test data
• Don't use deleteAll that affects other tests

Timing Issues

• setTimeout/sleep instead of condition-based waiting
• Not waiting for page stability
• Animations/transitions not complete

Fix patterns:

• Use waitForPageStability() helper
• Wait for specific DOM states
• Use Playwright's built-in waiting mechanisms

Linting Considerations

When fixing e2e tests, be aware of these eslint rules:

• playwright/no-networkidle - Avoid waitForLoadState('networkidle') (use condition-based waiting instead)
• payload/no-wait-function - Avoid custom wait() functions (use Playwright's built-in waits)
• payload/no-flaky-assertions - Avoid non-retryable assertions
• playwright/prefer-web-first-assertions - Use built-in Playwright assertions

Existing code may violate these rules - when adding new code, follow the rules even if existing code doesn't.

Verification

After fixing:

# Ensure dev server is running on port 3000
# Run test multiple times to confirm stability
for i in {1..10}; do
  pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" || break
done

# Run full suite
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts

# If you modified bundled code, test with prod build
lsof -ti:3000 | xargs kill -9 2>/dev/null
pnpm prepare-run-test-against-prod
pnpm dev:prod $SUITE_NAME
until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts