Flaky Test Catcher

Flaky Test Catcher — Analysis Protocol

1. Overview / Purpose

This skill defines the end-to-end protocol for detecting broken and flaky acceptance tests by analyzing recent CI failures on main, then opening structured GitHub issues so that automated remediation workflows can address the root causes.

You must follow this protocol strictly. Do not improvise or skip steps.

2. Inputs from pre-activation context

The workflow pre-activation step has already computed all run-level data. Do not re-query GitHub for run lists or issue counts. Use the values injected into your prompt:

Parse failed_run_ids as a JSON array immediately. Example: ["12345678","87654321"].

3. Fetching job logs

For each run ID in failed_run_ids:

3.1 List jobs for the run

gh api /repos/{owner}/{repo}/actions/runs/{run_id}/jobs?per_page=100

Replace {owner} and {repo} with the values from the repository's GitHub remote (visible in git remote get-url origin).

3.2 Filter to relevant failing jobs

From the returned jobs array, keep only jobs where both conditions hold:

• conclusion == "failure"
• The job name contains Matrix Acceptance Test (this is the job group that runs acceptance tests)

Ignore infrastructure jobs (e.g. lint, build, generate) — they do not produce --- FAIL: lines.

3.3 Fetch the log for each failing job

The GitHub API log endpoint returns a ZIP archive. Use the gh CLI to stream plain-text logs for a job:

gh run view --job {job_id} --log | grep '^--- FAIL:'

Log size warning: Job logs can be very large (10 MB+). Do not load the full log into context. Instead:

• Stream the log output through grep so only matching lines are retained.
• Scan only for lines matching the --- FAIL: pattern (see §4).
• Use grep -B3 -A3 or similar when you also need surrounding context for the "Sample Failure Output" issue section.
• Capture a small surrounding context (3–5 lines before/after each --- FAIL: line) for the "Sample Failure Output" issue section.

3.4 Pagination

If a job list response has 100 items and there may be more, check the Link header for a next page URL and repeat the request. In practice, a single run rarely has more than 100 jobs.

4. --- FAIL: extraction pattern

Scan each log for lines that match this exact pattern:

^--- FAIL: TestName (timing)

Examples of matching lines:

--- FAIL: TestAccResourceAgentConfiguration_alternateEnvironment (12.34s)
--- FAIL: TestAccSomeResource_basic (0.45s)

Rules:

• The line must start with --- FAIL: (three hyphens, a space, FAIL:, a space).
• The test name follows Go test naming: TestAcc... with optional underscore-separated sub-name.
• Extract only the test function name — strip the timing suffix (12.34s).
• Ignore bare FAIL lines without the --- prefix; those are package-level failure markers, not individual test failures.

Collect all extracted test names across all runs and all jobs. A test may appear multiple times (once per run where it failed) — track counts.

Same-run deduplication: If the same test name appears in multiple failing jobs within a single run (e.g. multiple shards both failing the same test), count it only once for that run. Deduplication is by run ID, not job ID — use a set per run when accumulating test names.

5. Fail-rate formula and thresholds

For each unique test name:

fail_rate = fail_count / total_run_count

Where:

• fail_count = number of distinct run IDs in which this test appeared as --- FAIL:
• total_run_count = the value from pre-activation context (already excludes cancelled runs)

6. Base-test-name grouping rule

Extract the base test name from each test function name using this rule:

Take the substring from the beginning up to (but not including) the first underscore _.

Pattern: TestAcc[^_]+

Examples:

One issue per base test name. All scenario variants (subtests/suffixes) belonging to the same base name are consolidated into a single issue. List each specific variant inside the issue body.

Fallback for non-TestAcc tests: All acceptance tests in this project follow the TestAcc prefix convention. If a non-TestAcc test name appears in the logs, treat everything up to the first _ (or the full name if no _) as the base name.

7. Commit analysis steps

For each base test name that will receive an issue, investigate whether any recent commit may already address the failure:

7.1 Find the oldest failing run timestamp

From the failed_run_ids list, identify the oldest run's created_at timestamp. You can get metadata for a single run:

gh api /repos/{owner}/{repo}/actions/runs/{run_id}

Take the minimum created_at across all failed runs.

7.2 Fetch commits on main since that timestamp

gh api "/repos/{owner}/{repo}/commits?sha=main&since={timestamp}&per_page=50"

Replace {timestamp} with the ISO 8601 value from step 7.1 (e.g. 2024-01-15T12:00:00Z).

7.3 For each commit, check relevance

For each commit returned:

a. Commit message relevance — does it reference any of:

• The base test name (e.g. TestAccResourceAgentConfiguration)
• The resource name (derive from the test name, e.g. agent_configuration → AgentConfiguration)
• Keywords: fix, flaky, test, revert

b. Changed file relevance — fetch the full commit detail to get changed file paths:

gh api /repos/{owner}/{repo}/commits/{sha}

Check if any file in files[].filename matches patterns like:

• *_test.go files whose name contains a token from the resource name
• Files in the same Go package directory as the test