CI Test Failures

CI Test Failure Diagnosis and Issue Filing

Recipe: Create an Issue for a Test Failure

When the user asks to create an issue for a failing test, follow these steps. Always redirect full output to a log file (not tail) so you can inspect it if the command fails.

Step 1: List failed tests (if user didn't specify one)

Omit --test to discover all failures. Redirect output to a log file:

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "<the-url-the-user-gave>" \
  --output /tmp/cfti-result.json \
  > /tmp/cfti-list.log 2>&1

dotnet run --project tools/CreateFailingTestIssue -- `
  --url "<the-url-the-user-gave>" `
  --output $env:TEMP/cfti-result.json `
  > $env:TEMP/cfti-list.log 2>&1

Then read the result with jq:

jq '{ success, availableFailedTests: .diagnostics.availableFailedTests, errorMessage: .errorMessage }' /tmp/cfti-result.json

Get-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='availableFailedTests';E={$_.diagnostics.availableFailedTests}}

If success is false, inspect the full log: cat /tmp/cfti-list.log (bash) or Get-Content $env:TEMP/cfti-list.log (PowerShell).

Ask the user which test to file for, then proceed to Step 2.

Step 2: Create the issue

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "<the-url-the-user-gave>" \
  --test "<test-name>" \
  --create \
  --output /tmp/cfti-result.json \
  > /tmp/cfti-create.log 2>&1

dotnet run --project tools/CreateFailingTestIssue -- `
  --url "<the-url-the-user-gave>" `
  --test "<test-name>" `
  --create `
  --output $env:TEMP/cfti-result.json `
  > $env:TEMP/cfti-create.log 2>&1

Then read the result:

jq '{ success, issue: .issue.createdIssue, errorMessage: .errorMessage }' /tmp/cfti-result.json

Get-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='issue';E={$_.issue.createdIssue}}

If success is false, inspect the full log: cat /tmp/cfti-create.log (bash) or Get-Content $env:TEMP/cfti-create.log (PowerShell).

That's it — do not add analysis comments, do not use --dry-run unless the user explicitly asks for a preview.

Rules:

• Always use --output <file> to keep JSON clean. Do NOT try to parse JSON from stdout — it is interleaved with dotnet build progress output.
• Use jq to extract fields from the output file. Key paths:
  • .success — whether the operation succeeded
  • .issue.createdIssue.number and .issue.createdIssue.url — the created/updated issue
  • .diagnostics.availableFailedTests[] — test names when --test is omitted
  • .errorMessage — error details when .success is false
• Do NOT create issues manually with gh issue create. The tool handles everything: resolving the run, finding the test, generating a template-compliant body, and creating the issue.
• Do NOT invent your own issue markdown. The tool generates content that matches .github/ISSUE_TEMPLATE/50_failing_test.yml.
• If the tool fails, report the error from diagnostics.log and the JSON output. Do not fall back to manual issue creation.

Recipe: Investigate a Failing Run (No Issue Creation)

To download and inspect failure artifacts without creating an issue:

cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Then search the downloaded logs and .trx files for errors.

Reference Documentation

Everything below is reference material for edge cases and deeper investigation.

Overview

Use this skill in two phases:

1. Investigate the run with DownloadFailingJobLogs.cs to fetch failed job logs and artifacts.
2. Create or update a failing-test issue with tools/CreateFailingTestIssue --create.

Tools covered

Quick Start

Step 1: Find the Run ID

Get the run ID from the GitHub Actions URL or use the gh CLI:

# From URL: https://github.com/microsoft/aspire/actions/runs/19846215629
#                                                        ^^^^^^^^^^
#                                                        run ID

# Or find the latest run on a branch
gh run list --repo microsoft/aspire --branch <branch-name> --limit 1 --json databaseId --jq '.[0].databaseId'

# Or for a PR
gh pr checks <pr-number> --repo microsoft/aspire

# From URL: https://github.com/microsoft/aspire/actions/runs/19846215629
#                                                        ^^^^^^^^^^
#                                                        run ID

# Or find the latest run on a branch
gh run list --repo microsoft/aspire --branch <branch-name> --limit 1 --json databaseId --jq '.[0].databaseId'

# Or for a PR
gh pr checks <pr-number> --repo microsoft/aspire

Step 2: Run the Tool

cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>

Example:

dotnet run DownloadFailingJobLogs.cs -- 19846215629

Step 3: Analyze Output

The tool creates files in your current directory:

What the Tool Does

1. Finds all failed jobs in a GitHub Actions workflow run
2. Downloads job logs for each failed job
3. Extracts test failures and errors from logs using regex patterns
4. Determines artifact names from job names (pattern: logs-{testShortName}-{os})
5. Downloads test artifacts containing .trx files and test logs
6. Extracts artifacts to local directories for inspection

Creating or Updating Failing-Test Issues

After you know which test failed, use the branch automation to create a failing-test issue in the known-issues format.

Preferred path: /create-issue from a PR or issue comment

Comment on the PR or issue with:

/create-issue --test "<test-name>" [--url <pr|run|job-url>] [--workflow <selector>] [--force-new]

Examples:

/create-issue --test "Tests.Namespace.Type.Method(input: 1)"
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/123
/create-issue "Tests.Namespace.Type.Method(input: 1)" https://github.com/microsoft/aspire/actions/runs/123/job/456
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/123/attempts/2/job/456?pr=321 --force-new

Notes:

• When the command is posted on a PR and no --url is supplied, the workflow defaults to that PR URL.
• --workflow defaults to ci.
• --force-new bypasses issue reuse and always requests a fresh issue.
• The workflow requires write or admin access to the repository before it will create or update issues.

Supported source URLs

The resolver accepts:

• Pull request URLs: https://github.com/<owner>/<repo>/pull/<number>
• Workflow run URLs: https://github.com/<owner>/<repo>/actions/runs/<run-id>
• Attempt URLs: https://github.com/<owner>/<repo>/actions/runs/<run-id>/attempts/<attempt>
• Job URLs: https://github.com/<owner>/<repo>/actions/runs/<run-id>/job/<job-id>
• Attempt job URLs with query strings

Local path: run the resolver directly

Always use --output to write results to a file so JSON is not interleaved with build output:

To generate the JSON result locally without creating an issue (dry run):

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "https://github.com/microsoft/aspire/actions/runs/123" \
  --test "<test-name>" \
  --repo "microsoft/aspire" \
  --output /tmp/cfti-result.json

dotnet run --project tools/CreateFailingTestIssue -- `
  --url "https://github.com/microsoft/aspire/actions/runs/123" `
  --test "<test-name>" `
  --repo "microsoft/aspire" `
  --output $env:TEMP/cfti-result.json

To resolve the failure and create the issue on GitHub in one step:

dotnet run --project tools/CreateFailingTestIssue -- \
  --url "https://github.com/microsoft/aspire/actions/runs/123" \
  --test "<test-name>" \
  --repo "microsoft/aspire" \
  --create \
  --output /tmp/cfti-result.json

dotnet run --project tools/CreateFailingTestIssue -- `
  --url "https://github.com/microsoft/aspire/actions/runs/123" `
  --test "<test-name>" `
  --repo "microsoft/aspire" `
  --create `
  --output $env:TEMP/cfti-result.json

Read the result with jq:

jq '{ success, issue: .issue, availableFailedTests: .diagnostics.availableFailedTests }' /tmp/cfti-result.json

Get-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, @{N='issue';E={$_.issue}}, @{N='availableFailedTests';E={$_.diagnostics.availableFailedTests}}

If --test is omitted, the tool emits structured JSON for all failing tests it found in the run (useful for picking which test to file).

The command writes a diagnostics.log file in the current directory. The JSON output (written to the --output file or stdout) contains:

• the resolved run and job URLs
• either the matched canonical and display test names plus generated issue content, or a per-test list of all failures in the run
• primary failure details (error, stack trace, stdout)
• when --create is set, the created issue number and URL
• warnings and alternate failed-test names if the match fails