Pester Failure Analysis

Analyze Pester Test Failures

Investigate and troubleshoot Pester test failures in GitHub Actions workflows. Understand what tests are failing, why they're failing, and provide recommendations for test fixes.

When to Use This Skill

Use this skill when you need to:

• Understand why a specific Pester test is failing
• Interpret test failure messages and error output
• Analyze test result data from CI workflow runs (XML, logs, stack traces)
• Identify the root cause of test failures (test logic, assertion failure, exception, timeout, skip/ignore reason)
• Provide recommendations for fixing failing tests
• Compare expected vs. actual test behavior
• Debug test environment issues (missing dependencies, configuration problems)
• Understand test skip/ignored/inconclusive status reasons

Do not use this skill for:

• General PowerShell debugging unrelated to tests
• Test infrastructure/CI setup issues (except as they affect test failure interpretation)
• Performance analysis or benchmarking (that's a different investigation)

Quick Start

⚠️ CRITICAL: The Workflow Must Be Followed IN ORDER

This skill describes a sequential 6-step analysis workflow. Skipping steps or jumping around leads to incomplete analysis and incorrect conclusions.

The Problem: It's easy to skip to Step 4 or 5 without doing Steps 1-2, resulting in missing data and bad conclusions.

The Solution: Use the automated analysis script to enforce the workflow:

# Automatically runs Steps 1-6 in order, preventing skipping
./.github/skills/analyze-pester-failures/scripts/analyze-pr-test-failures.ps1 -PR <PR_NUMBER>

# Example:
./.github/skills/analyze-pester-failures/scripts/analyze-pr-test-failures.ps1 -PR 26800

This script:

1. ✓ Fetches PR status automatically
2. ✓ Downloads artifacts (can't skip, depends on Step 1)
3. ✓ Extracts failures (can't skip, depends on Step 2)
4. ✓ Analyzes error messages
5. ✓ Documents context
6. ✓ Generates recommendations

Only use the manual commands below if you fully understand the workflow.

Manual Workflow (for reference)

# Step 1: Identify the failing job
gh pr view <PR_NUMBER> --json 'statusCheckRollup' | ConvertFrom-Json | Where-Object { $_.conclusion -eq 'FAILURE' }

# Step 2: Download artifacts (extract RUN_ID from Step 1)
gh run download <RUN_ID> --dir ./artifacts
gh run view <RUN_ID> --log > test-logs.txt

# Step 3-6: Extract, analyze, and interpret
# (See Analysis Workflow section below)

Common Test Failure Analysis Approaches

1. Interpreting Assertion Failures

The most common test failure is when an assertion doesn't match expectations.

Example:

Expected $true but got $false at /path/to/test.ps1:42
Assertion failed: Should -Be "expected" but was "actual"  

How to analyze:

• Read the assertion message: what was expected vs. what was actual?
• Check the test logic: is the expectation correct?
• Look for mock/stub issues: are dependencies configured correctly?
• Check parameter values: what inputs were passed to the function under test?

2. Exception Failures

Tests fail when PowerShell throws an exception instead of successful completion.

Example:

Command: Write-Host $null
Error: Cannot bind argument to parameter 'Object' because it is null.

How to analyze:

• Read the exception message: what operation failed?
• Check the stack trace: where in the test or tested code did it throw?
• Verify preconditions: does the test setup provide required values/mocks?
• Look for environmental issues: missing modules, permissions, file system state?

3. Timeout Failures

A test takes longer than the allowed timeout to complete.

Example:

Test 'Should complete in reasonable time' timed out after 30 seconds

How to analyze:

• Is the timeout appropriate for this test type? (network tests need more time)
• Is there an infinite loop in the test or tested code?
• Are there resource contention issues on the CI runner?
• Does the test hang waiting for something (file lock, network, process)?

4. Skip/Ignored Reason Analysis

Tests marked as skipped or ignored provide clues about test environment.

Example:

Test marked [Skip("Only runs on Windows")] - running on Linux
Test marked [Ignore("Known issue #12345")]

How to analyze:

• Read the skip/ignore reason: is it still valid?
• Check if environment has changed: platform, module versions, etc.
• Verify issue status: is the known issue still open? Has it been fixed?
• Determine if skip should be removed or if test needs environment changes

5. Flaky/Intermittent Failures

Tests that sometimes pass, sometimes fail indicate race conditions or environment sensitivity.

Example:

• Test passes locally but fails on CI
• Test passes first run of suite, fails on second run
• Test passes on Windows but fails on Linux

How to analyze:

• Look for timeout races: is timing involved in the test?
• Check for test isolation issues: does one test affect another?
• Verify environment differences: CI vs. local paths, permissions, versions
• Look for external dependencies: network calls, file I/O, process interactions

Key Artifacts and Locations

Analysis Workflow

⚠️ Important: These Steps MUST Be Followed In Order

Each step depends on the previous one. Skipping or re-ordering steps causes incomplete analysis:

• Step 1 (identify jobs) → You get the RUN_ID needed for Step 2
• Step 2 (download) → You get the artifacts needed for Step 3
• Step 3 (extract) → You discover what failures exist for Step 4
• Step 4 (read messages) → You understand the errors to analyze in Step 5
• Step 5 (context) → You gather information to make recommendations in Step 6
• Step 6 (interpret) → You use all above to recommend fixes

Real Problem We Had:

• ❌ Jumped to Step 3 without Step 1-2
• ❌ Used random test data from context instead of downloading PR artifacts
• ❌ Skipped Steps 5-6 entirely
• ❌ Made recommendations without full context

Result: Wrong analysis and recommendations that didn't actually fix the problem.

Recommended: Use the Automated Script

./.github/skills/analyze-pester-failures/scripts/analyze-pr-test-failures.ps1 -PR <PR_NUMBER>

This enforces the workflow and prevents skipping.

Step 2: Get Test Results

Fetch the test result artifacts and job logs:

# Download artifacts including test XML results
gh run download <RUN_ID> --dir ./artifacts

# Get job logs
gh run view <RUN_ID> --log > test-logs.txt

# Inspect test XML
$xml = [xml](Get-Content ./artifacts/junit-pester-*.xml)
$xml.'test-results' | Select-Object total, failures, errors, ignored, inconclusive

Step 3: Extract Specific Failures

Find the failing test cases in the XML:

# Get all failed test cases
$xml = [xml](Get-Content ./artifacts/junit-pester-*.xml)
$failures = $xml.SelectNodes('.//test-case[@result = "Failure"]')

# For each failure, display key info
$failures | ForEach-Object {
    [PSCustomObject]@{
        Name = $_.name
        Description = $_.description
        Message = $_.failure.message
        StackTrace = $_.failure.'stack-trace'
    }
}

Step 4: Read the Error Message

The error message tells you what went wrong:

Assertion failures:

Expected $true but got $false
Expected "value1" but got "value2"
Expression should have failed with exception, but didn't

Exceptions:

Cannot find a parameter with name 'Name'
Property 'Property' does not exist on 'Object'
Cannot bind argument to parameter because it is null

Timeouts:

Test timed out after 30 seconds
Test is taking too long to complete

Step 5: Understand the Context

Look at the test file to understand what was being tested:

# Find the test file mentioned in the stack trace
# Example: /path/to/test/Feature.Tests.ps1:42

# Read the test code around that line
code <test-file-path>:<line-number>

# Understand:  
# - What assertion is on that line?
# - What is the test trying to verify?
# - What are the setup/mock/before conditions?
# - Are there recent changes to the function being tested?

Step 6: Interpret the Failure

Determine the root cause category:

Test issue (needs code fix):

• Assertion logic is wrong
• Test expectations don't match actual behavior
• Test setup is incomplete
• Mock/stub configuration missing

Environmental issue (needs environment change):

• Test assumes a specific file or registry entry exists
• Test requires Windows/Linux specifically
• Test requires specific PowerShell version
• Test requires specific module version
• Timing-sensitive test affected by CI load

Data issue (needs input data change):

• Test data no longer valid
• External API changed format
• Configuration file has changed structure

Flakiness (needs test hardening):

• Race condition in test
• Timing assumptions too tight
• Resource contention on CI runner
• Non-deterministic behavior in tested code

Common Test Failure Patterns