New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Official agent skill

Verify Tests Catch the Bug

by dotnet in dotnet/maui

Confirms that newly added tests actually fail without the fix, auto-detecting UI, device, unit or XAML tests and running the matching runner.

OfficialMITAuto-check passedTesting & QA

Install Verify Tests Catch the Bug

skills CLI
$ npx skills add dotnet/maui --skill verify-tests-fail-without-fix -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install dotnet/maui verify-tests-fail-without-fix --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/dotnet/maui.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.github/skills/verify-tests-fail-without-fix .claude/skills/verify-tests-fail-without-fix && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
verify-tests-fail-without-fix
GitHub stars
23k
Token cost
~2.7k tokens
SKILL.md length
999 words
Files
5 (incl. scripts)
Skills in repo
27
Repo updated
First seen
Licence
MIT

At a glance

Confirms that newly added tests actually fail without the fix, auto-detecting UI, device, unit or XAML tests and running the matching runner.

  • Works in 4 steps: Determine Mode → Construct Command → Interpret Results → …
  • A PR adds tests for a bug and you need proof that they fail without the fix
  • SKILL.md covers Supported Test Types, Activation Guard, ⚠️ CRITICAL: Inverted… and Workflow, plus 8 more sections
  • Runs PowerShell scripts from its folder; calls git, pwsh and dotnet

What it does

The skill checks that tests reproduce a bug, so a test that passes without the fix counts as a failed verification. It detects the test type from the changed paths (UI tests, device tests, unit tests or XAML unit tests), picks the matching runner, and needs a platform for UI and device tests plus either test files in the PR or an explicit test filter. It does not write tests, run tests with no verification context or review code.

Two modes exist: verify failure only when the PR has no fix files, and full verification when it does, which validates the fix as well. The script verify-tests-fail.ps1 handles every fix-file transition and cleanup, so the worktree should not be edited by hand around it. The result meaning is inverted on purpose: failing tests without the fix are good, passing ones are bad, and the agent must never call that second case a pass. If you only ask how to read the output, it explains the result contract without running anything.

When your agent uses it

  • A PR adds tests for a bug and you need proof that they fail without the fix
  • Checking whether a reproduction test really detects the reported issue
  • Validating both a new test and its accompanying fix in one run

Example prompts

  • “Verify that the new Button tests in this PR fail without the fix on Android.”
  • “Run full verification on this PR for the iOS platform.”
  • “What does it mean when tests pass without the fix?”

Requirements

  • git
  • PowerShell
  • .NET SDK
  • Compatibility (from SKILL.md): Requires git, PowerShell, and .NET SDK for building and running tests.

Workflow steps

4 steps, taken from the step headings in SKILL.md.

  1. Determine Mode
  2. Construct Command
  3. Interpret Results
  4. Report

What it can do on your machine

Read from SKILL.md and the folder at commit b926f05. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 2 files in scripts/ (PowerShell), which the agent can run.

    Shell commands in SKILL.md call:

    • git
    • pwsh
    • dotnet
    • gh

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md. Its commands use git and gh, which can reach the network depending on how they are called.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Requires git, PowerShell, and .NET SDK for building and running tests.

    From compatibility in the SKILL.md frontmatter.

Context cost

Verify Tests Catch the Bug loads about 2.7k tokens when it runs. Until then it costs about 66 tokens; SKILL.md has 999 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~66
When it runs · the whole SKILL.md, loaded when a task matches
~2.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from dotnet/maui at commit b926f05, republished under its MIT licence (© dotnet). 999 words, ~2,668 tokens.

Download SKILL.mdSave it as .claude/skills/verify-tests-fail-without-fix/SKILL.md (or your agent's skills folder). This skill also uses 4 other files; get the full folder from GitHub.
name
verify-tests-fail-without-fix
description
Verifies tests catch the bug. Auto-detects test type (UI tests, device tests, unit tests) and dispatches to the appropriate runner. Supports two modes - verify failure only (test creation) or full verification (test + fix validation).
compatibility
Requires git, PowerShell, and .NET SDK for building and running tests.
metadata.author
dotnet-maui
metadata.version
2.0

Verify Tests Fail Without Fix

Verifies tests actually catch the issue. Supports all test types (UI tests, unit tests, XAML tests, device tests) and two workflow modes.

Supported Test Types

Test TypeAuto-Detected FromRunner
UITestTestCases.Shared.Tests/, TestCases.HostApp/BuildAndRunHostApp.ps1
DeviceTestDeviceTests/Run-DeviceTests.ps1
UnitTest*.UnitTests/, Graphics.Tests/dotnet test
XamlUnitTestXaml.UnitTests/dotnet test

Test type is auto-detected from changed files. Override with -TestType if needed.

-Platform is required for UI and Device tests. It selects which platform to verify the fix on. Unit and XAML tests do not require -Platform.

Activation Guard

🛑 This skill ONLY verifies that existing tests reproduce a bug. Do NOT activate for:

  • Writing new tests → use write-tests-agent
  • Running tests without verification context → use run-device-tests
  • Code review → use code-review skill
  • General test advice

Requires: a platform and either test files in the PR or an explicit TestFilter.

⚠️ CRITICAL: Inverted Pass/Fail Semantics

In this skill, test outcomes mean the OPPOSITE of normal:

Test Result (without fix)Verification ResultWhy
Tests FAIL✅ GOODTests detect the bug
Tests PASS❌ BADTests miss the bug

NEVER say "verification passed" when tests PASS without the fix.

Workflow

Interpretation-Only Requests

If the caller asks how to read or report verification output, explain the result contract below without running the script. Do not turn a question about result semantics into a verification run.

Step 1: Determine Mode
  • Check if fix files exist in the PR (non-test code changes detected by the script from the git diff)
  • If fix files present → Full Verification mode (-RequireFullVerification)
  • If no fix files → Verify Failure Only mode (omit the flag)
Step 2: Construct Command
powershell
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 `
  -Platform <platform> `
  -TestFilter "<filter>" `
  [-RequireFullVerification]  # Only if fix files exist

Run the prescribed script once and let it own every fix-file transition and cleanup step. Do not manually mutate the worktree before or after it with git checkout, git clean, git restore, git reset, git stash, git apply --reverse/git apply -R, or an equivalent file-reversion command. If the script leaves an unexpected tracked or untracked change, report the verification as Blocked with the observed status instead of cleaning it up.

Step 3: Interpret Results

⚠️ Remember: test outcomes are INVERTED from normal!

  • If the command continues in the background and returns a shell/session ID, call the matching result-read tool with that exact ID. If it is still running, keep waiting on the same ID until it completes. Never report, summarize partial output, or end the turn before observing the completed result and its VERIFICATION PASSED, VERIFICATION FAILED, or error/timeout outcome.
  • Script outputs VERIFICATION PASSED → Tests catch the bug ✅
  • Script outputs VERIFICATION FAILED → Tests don't catch the bug ❌
  • Script outputs error/timeout → Report as Blocked
Step 4: Report
  • Always report the script's exact terminal marker and explain the observed phase results:
    • VERIFICATION PASSED in failure-only mode means the test failed without the fix, proving that it catches the bug.
    • VERIFICATION PASSED in full mode means the test failed without the fix and passed with the fix.
    • VERIFICATION FAILED because the test passed without the fix means the test does not catch the bug.
    • VERIFICATION FAILED because the test failed with the fix means the fix did not resolve the tested behavior, or the test failed for another reason.
    • An error, timeout, or unexpected worktree change is Blocked; report the observed evidence without converting it into a pass or cleaning it up.

Mode 1: Verify Failure Only (Test Creation)

Use when creating tests before writing a fix:

  • Runs tests to verify they FAIL (proving they catch the bug)
  • No fix files required
  • Perfect for test-first development
bash
# Auto-detect test type and filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android

# Explicit test type + filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -TestType UnitTest -TestFilter "Maui12345"
Show full SKILL.md (423 more words)Show less

Mode 2: Full Verification (Fix Validation)

Use when validating both tests and fix:

  1. Without fix - tests should FAIL (bug is present)
  2. With fix - tests should PASS (bug is fixed)
bash
# Auto-detect everything (recommended)
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform android -RequireFullVerification

# With explicit test filter
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 -Platform ios -TestFilter "Issue33356" -RequireFullVerification

Note: -RequireFullVerification ensures the script errors if no fix files are detected, preventing silent fallback to failure-only mode.

Requirements

Verify Failure Only Mode:

  • Test files in the PR (or working directory)

Full Verification Mode:

  • Test files in the PR
  • Fix files in the PR (non-test code changes)

The script auto-detects which mode to use based on whether fix files are present.

Expected Output

Verify Failure Only Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  Tests FAILED as expected!                                ║
║  This proves the tests correctly reproduce the bug.       ║
╚═══════════════════════════════════════════════════════════╝

Full Verification Mode:

╔═══════════════════════════════════════════════════════════╗
║              VERIFICATION PASSED ✅                       ║
╠═══════════════════════════════════════════════════════════╣
║  - FAIL without fix (as expected)                         ║
║  - PASS with fix (as expected)                            ║
╚═══════════════════════════════════════════════════════════╝

What It Does

Verify Failure Only Mode (no fix files):

  1. Fetches base branch from origin (if available)
  2. Auto-detects test type from changed files (UITest, UnitTest, XamlUnitTest, DeviceTest)
  3. Auto-detects test classes from changed test files
  4. Routes to the appropriate test runner
  5. Runs tests (should FAIL to prove they catch the bug)
  6. Reports result

Full Verification Mode (fix files detected):

  1. Fetches base branch from origin to ensure accurate diff
  2. Auto-detects fix files (non-test code) from git diff
  3. Auto-detects test type and test classes from changed files
  4. Reverts fix files to base branch
  5. Runs tests using the appropriate runner (should FAIL without fix)
  6. Restores fix files
  7. Runs tests using the appropriate runner (should PASS with fix)
  8. Generates markdown reports:
    • CustomAgentLogsTmp/TestValidation/verification-report.md - Full detailed report
    • CustomAgentLogsTmp/PRState/verification-report.md - Validate section for agent
  9. Reports result

Note: PR label management (s/ai-reproduction-confirmed / s/ai-reproduction-failed) is handled by Review-PR.ps1, not by this script.

Output Files

The skill generates output files under CustomAgentLogsTmp/PRState/<PRNumber>/PRAgent/gate/verify-tests-fail/:

FileDescription
verification-report.mdComprehensive markdown report with test results and full logs
verification-log.txtText log of the verification process
test-without-fix.logFull test output from run without fix
test-with-fix.logFull test output from run with fix

Plus test logs in CustomAgentLogsTmp/:

  • UITests/ - UI test device logs and output
  • DeviceTests/ - Device test output
  • UnitTests/ - Unit test output

Example structure:

CustomAgentLogsTmp/
├── UITests/                           # UI test logs
│   ├── android-device.log
│   └── test-output.log
├── DeviceTests/                       # Device test logs
│   └── test-output.log
├── UnitTests/                         # Unit/XAML test logs
│   └── test-output.log
└── PRState/
    └── 27847/
        └── PRAgent/
            └── gate/
                └── verify-tests-fail/
            ├── verification-report.md  # Full detailed report
            ├── verification-log.txt
            ├── test-without-fix.log
            └── test-with-fix.log

PR Number Detection:

  • Auto-detected from branch name (e.g., pr-27847)
  • Falls back to gh pr view command
  • Uses "unknown" if detection fails
  • Can be manually specified with -PRNumber parameter

Troubleshooting

ProblemCauseSolution
No fix files detectedBase branch detection failed or no non-test files changedUse -FixFiles or -BaseBranch explicitly
Tests pass without fixTests don't detect the bugReview test assertions, update test
Tests fail with fixFix doesn't work or test is wrongReview fix implementation
App crashesDuplicate issue numbers, XAML errorCheck device logs
Element not foundWrong AutomationId, app crashedVerify IDs match

Optional Parameters

bash
# Require full verification (fail if no fix files detected) - recommended
-RequireFullVerification

# Explicit test type (auto-detected if omitted)
-TestType UnitTest    # or XamlUnitTest, DeviceTest, UITest

# Explicit test filter
-TestFilter "Issue32030|ButtonUITests"

# Explicit fix files  
-FixFiles @("src/Core/src/File.cs")

# Explicit base branch (ordinary PR metadata remains authoritative)
-BaseBranch "main"

# Full commit SHA (frozen-fixture mode uses the local immutable diff)
-BaseBranch "$(git rev-parse HEAD^)"

© dotnet, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files (scripts) in .github/skills/verify-tests-fail-without-fix of dotnet/maui.

  • SKILL.md
  • scripts/Verify-TestsFail.Tests.ps1
  • scripts/verify-tests-fail.ps1
  • tests/eval.protocol.vally.yaml
  • tests/eval.vally.yaml

Open the folder on GitHubat commit b926f05

Compare with similar skills

Verify Tests Catch the Bug next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Verify Tests Catch the Bug compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Verify Tests Catch the Bug this skilldotnet/maui23k—~2.7kAutomated safety check: PassMIT
Analyze Azdo BuildDataDog/dd-trace-dotnet573—~3.5kAutomated safety check: PassApache-2.0
Pester Failure AnalysisPowerShell/PowerShell56k—~5.1kAutomated safety check: PassMIT
TiDB Test Diff Triagepingcap/tidb41k—~498Automated safety check: PassApache-2.0
OpenLogi Change VerificationAprilNEA/OpenLogi23k—~1.4kAutomated safety check: PassApache-2.0
Macios CI Failure Inspectordotnet/macios2.9k—~2.3kAutomated safety check: PassCustom licence

Similar skills

  • Analyze Azdo Build

    DataDog/dd-trace-dotnet

    Official

    Analyze Azure DevOps CI build failures in dd-trace-dotnet pipeline.

    573 GitHub stars~3.5k tokensUpdated today
    Testing & QAAuto-check passed

  • Pester Failure Analysis

    PowerShell/PowerShell

    Investigates failing Pester tests in PowerShell CI jobs by following a six-step workflow from pull request status to documented fix recommendations.

    56k GitHub stars~5.1k tokensUpdated today
    Testing & QAAuto-check passed

  • TiDB Test Diff Triage

    pingcap/tidb

    Investigates TiDB plan or test-result diffs that the change does not explain, ruling out failpoint setup and merge effects before expected outputs are updated.

    41k GitHub stars~498 tokensUpdated today
    Testing & QAAuto-check passed

  • OpenLogi Change Verification

    AprilNEA/OpenLogi

    Plans the smallest check that could disprove a code change in the OpenLogi project, then escalates through reproduction, focused tests and a final gate before a push.

    23k GitHub stars~1.4k tokensUpdated 5 days ago
    Testing & QAAuto-check passed

  • Macios CI Failure Inspector

    dotnet/macios

    Official

    Investigate and triage CI failures for dotnet/macios from Azure DevOps build URLs.

    2.9k GitHub stars~2.3k tokensUpdated today
    Testing & QAAuto-check passed

  • Designing Tests

    CloudAI-X/claude-workflow-v2

    Designs and implements testing strategies for any codebase. An agent skill from CloudAI-X/claude-workflow-v2.

    1.4k GitHub starsUsed in 1 repo~1.5k tokens
    Testing & QAAuto-check passed

More from dotnet/maui

All 27 skills in this repo

  • Copilot Session Failure Analysis

    dotnet/maui

    Official

    Mines local Copilot CLI session logs for dotnet/maui to rank costly or failing runs, tag recurring failure modes, propose repo edits and emit guard evals.

    23k GitHub stars~3.4k tokensUpdated today
    Auto-check passed

  • Evaluate PR Tests

    dotnet/maui

    Official

    Reviews the tests added in a pull request for fix coverage, quality, edge cases and test type, and recommends lighter test types where they would do.

    23k GitHub stars~2.9k tokensUpdated today
    Auto-check passed

  • .NET MAUI Release Readiness

    dotnet/maui

    Official

    Produces evidence-backed ship-readiness verdicts for .NET MAUI Servicing Releases and Previews, and drafts public-safe release handoff pages from the result.

    23k GitHub stars~15k tokensUpdated today
    Auto-check passed

  • MAUI PR Performance Analysis

    dotnet/maui

    Official

    Interprets pinned managed benchmark evidence for a dotnet/maui pull request and writes a narrative for the performance review workflow, without running or publishing anything.

    23k GitHub stars~2.4k tokensUpdated today
    Auto-check passed

  • PR Finalize

    dotnet/maui

    Official

    Checks that a pull request's title and description match its implementation and reviews the code for best practices before merge, without posting anything.

    23k GitHub stars~3.1k tokensUpdated today
    Auto-check passed

  • .NET MAUI Dependency Flow Guide

    dotnet/maui

    Official

    Adds MAUI-specific guardrails on top of the maestro-cli skill and Maestro MCP tools for darc, BAR, and channel or feed lookups in dotnet/maui.

    23k GitHub stars~10k tokensUpdated today
    Auto-check passed

Works with

Categories

Questions about Verify Tests Catch the Bug

What does Verify Tests Catch the Bug do?

Confirms that newly added tests actually fail without the fix, auto-detecting UI, device, unit or XAML tests and running the matching runner. The skill checks that tests reproduce a bug, so a test that passes without the fix counts as a failed verification. It detects the test type from the changed paths (UI tests, device tests, unit tests or XAML unit tests), picks the matching runner, and needs a platform for UI and device tests plus either test files in the PR or an explicit test filter.

When should I use Verify Tests Catch the Bug?

Verify Tests Catch the Bug fits situations like: A PR adds tests for a bug and you need proof that they fail without the fix; checking whether a reproduction test really detects the reported issue; validating both a new test and its accompanying fix in one run.

How do I install Verify Tests Catch the Bug in Claude Code?

Run `npx skills add dotnet/maui --skill verify-tests-fail-without-fix -a claude-code`. Or copy the skill folder (.github/skills/verify-tests-fail-without-fix in dotnet/maui) into .claude/skills/verify-tests-fail-without-fix in your project. Claude Code loads it when a task matches its description.

How do I install Verify Tests Catch the Bug in Codex?

Run `npx skills add dotnet/maui --skill verify-tests-fail-without-fix -a codex`. Or copy the skill folder (.github/skills/verify-tests-fail-without-fix in dotnet/maui) into .agents/skills/verify-tests-fail-without-fix in your project. Codex loads it when a task matches its description.

Can I use Verify Tests Catch the Bug in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add dotnet/maui --skill verify-tests-fail-without-fix -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/verify-tests-fail-without-fix, .gemini/skills/verify-tests-fail-without-fix, .github/skills/verify-tests-fail-without-fix and .opencode/skills/verify-tests-fail-without-fix in your project.

What does Verify Tests Catch the Bug need to run?

Going by SKILL.md and its folder, Verify Tests Catch the Bug needs PowerShell for the scripts in its folder and the command-line tools its instructions call (git, pwsh, dotnet and gh). Our summary lists: git; PowerShell; .NET SDK. Compatibility (from SKILL.md): Requires git, PowerShell, and .NET SDK for building and running tests..

Does Verify Tests Catch the Bug access the network?

SKILL.md contains no URLs. Its commands use git and gh, which can reach the network depending on how they are called. This is read from the text; nothing was executed.

Is Verify Tests Catch the Bug safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Verify Tests Catch the Bug use?

Verify Tests Catch the Bug is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Verify Tests Catch the Bug use?

About 2.7k tokens (SKILL.md is roughly 11k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Verify Tests Catch the Bug?

Skills that share tags, products or a category with Verify Tests Catch the Bug: Analyze Azdo Build (DataDog/dd-trace-dotnet, 573 stars), Pester Failure Analysis (PowerShell/PowerShell, 56k stars), TiDB Test Diff Triage (pingcap/tidb, 41k stars) and OpenLogi Change Verification (AprilNEA/OpenLogi, 23k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Verify Tests Catch the Bug?

dotnet (a GitHub organization, an official publisher) maintains it in dotnet/maui, which has 23,321 GitHub stars. The repository holds 27 skills in this directory. The repository was last updated on October 8, 2026.

Source: dotnet/maui on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.