Testfx Acceptance Validation

3. Run a focused regression and verify execution

This Windows example runs one existing packed-package regression. It is a package layout check, not evidence of real UWP/WinUI activation. Adapt the project/filter and expected names for the changed contract. Use a unique results directory, retain logs/binlogs, and do not infer expectations from the report being verified.

# After the successful configuration-matched pack above.
$project = 'test\IntegrationTests\MSTest.Acceptance.IntegrationTests\MSTest.Acceptance.IntegrationTests.csproj'
$case = 'PackedMSTestTestAdapter_ContainsRequiredWindowsApplicationModelAssets'
$results = Join-Path (Get-Location).Path ('artifacts\TestResults\Release\acceptance-' + [Guid]::NewGuid().ToString('N'))
New-Item -ItemType Directory -Path $results | Out-Null

$packages = @(Get-ChildItem -LiteralPath artifacts\packages\Release\Shipping `
    -Filter 'MSTest.TestAdapter.*.nupkg' -File)
if ($packages.Count -ne 1) { throw "Expected one current adapter package, found $($packages.Count)." }
$packages | Get-FileHash -Algorithm SHA256 | ConvertTo-Json |
    Set-Content -LiteralPath "$results\package-provenance.json" -Encoding UTF8

& .\.dotnet\dotnet.exe build $project -c Release "-bl:$results\build.binlog"
if ($LASTEXITCODE -ne 0) { throw "Acceptance build failed: $LASTEXITCODE" }

$filter = "FullyQualifiedName~WindowsApplicationModelPackageTests.$case"
# Blank these global properties so discovery does not inherit execution-only reporters.
& .\.dotnet\dotnet.exe test --project $project -c Release --no-build `
    --list-tests --filter $filter -p:UsingDotNetTest=true `
    -p:TestingPlatformCommandLineArguments= -p:TestRunnerAdditionalArguments= `
    "-bl:$results\discovery.binlog" 2>&1 | Tee-Object -FilePath "$results\discovery.log"
$discoveryExit = $LASTEXITCODE
if ($discoveryExit -ne 0) { throw "Acceptance discovery failed: $discoveryExit" }
# Reconcile discovery.log with the one source-declared case before executing.

# The plan comes from source/data rows and independent discovery, not this run's TRX.
$expectedPath = Join-Path $results 'expected-tests.json'
ConvertTo-Json -InputObject @($case) | Set-Content -LiteralPath $expectedPath -Encoding UTF8
$started = [DateTimeOffset]::UtcNow.ToString('o')
& .\.dotnet\dotnet.exe test --project $project -c Release --no-build `
    --filter $filter `
    -p:UsingDotNetTest=true "-p:ArtifactsTestResultsDir=$results" `
    "-bl:$results\test.binlog" --show-test-results all `
    2>&1 | Tee-Object -FilePath "$results\execution.log"
$testExit = $LASTEXITCODE

$reports = @(Get-ChildItem -LiteralPath $results -Filter '*.trx' -File)
if ($reports.Count -ne 1) { throw "Expected one outer acceptance TRX, found $($reports.Count). Exit: $testExit" }
python .github\skills\testfx-acceptance-validation\scripts\verify_execution.py `
    --trx $reports[0].FullName --expected-tests $expectedPath `
    --started-after $started --process-exit-code $testExit
if ($LASTEXITCODE -ne 0) { throw 'Acceptance execution proof failed.' }

The verifier is standard-library Python and checks one outer acceptance TRX per module/TFM. The JSON plan is a nonempty array of exact TRX testName strings, including data-row display names and multiplicity. Establish them from source, data expansion, and separate discovery; reconcile discovery with the plan before execution. Neither --list-tests nor a list copied from a passing report proves execution. Record discovery's own exit/output and use the same selector for the run.

The output distinguishes planned, reported-selected, executed, passed, failed, and skipped counts. It requires all planned cases to be present and passed, consistent counters, a fresh run start, unique execution IDs, and outer exit zero. Zero tests, NotExecuted/inconclusive/skipped results, missing modules, stale artifacts, and a nonzero exit fail the gate. Run it separately for each planned module with that module's expected names; aggregate success only after every gate passes. Legitimate unrelated skips belong in a separate selection/report, not a lowered proof bar. For negative/discovery/help-only product commands, assert their contractual exits and output inside passing acceptance cases; do not pass their child TRX to this gate.

4. Prove the capable CI job will run the cases

Inspect azure-pipelines.yml and the actual referenced templates on the branch that will ship. Trace change detection/job conditions, SkipTests, configuration, project/module selection, filter/category, conditional attributes, and environment all the way to each consuming test step. A category missing from a new test means the dedicated filter will never select it.

For real Windows app-model changes:

• The blocking WindowsAppModel job uses eng\pipelines\steps\test-windows-app-model.yml, Release packs/builds, and eng\pipelines\test-windows-app-model-preflight.ps1. Preflight requires an interactive/elevated AppX-capable Windows session, appropriate VS/UWP tooling, and installed SDKs. A Windows image label alone is insufficient.
• Real activation classes use [TestCategory("WindowsApplicationModel")], TESTFX_RUN_WINDOWS_APP_MODEL_TESTS=1, and relevant OS/member conditions. WindowsApplicationModelPackageTests is instead selected by its own FQN filter; package-content checks do not substitute for activation.
• Native .NET 10 tests need TESTFX_DOTNET_10_PATH pointing to the isolated installation from that template, including the required .NET 8 runtime. Verify the variable on every consuming MSTest/MTP step, not just where the installation happens. Copy current pinned versions from the template, not this skill. The repository SDK and the isolated shipping SDK have different roles.
• WinApp interoperability additionally requires TESTFX_RUN_WINAPP_CLI_INTEROP_TESTS=1, TESTFX_WINAPP_CLI_PATH, and the pinned, hash-verified WinApp CLI. Do not enable it for unrelated changes.

Compare local planned identities/counts with CI's actual selection and executed outcomes. Read TRX/TestResults and the test-step log, not merely the job badge or successful PublishTestResults. Reconcile retries per attempt/module; do not sum duplicate retry reports. If the capable job is gated out or prerequisites are missing, report CI execution unproven/blocked even if a different job is green. Record build/job/attempt identity and diagnostic artifact location. Never fix that by weakening conditions or treating skipped regressions as validated.

Historical failure modes motivating these gates: missing category in #11614, 29 NotExecuted cases and missing consuming SDK path in #11805, and packed-host launch-profile/discovery routing in #11804. These are examples, not instructions to modify those PRs.

5. Preserve evidence and clean up only owned state

Retain exact commands and component versions, working directory/environment overrides (no secrets), exits, expected/observed identities and counts, pack hashes, resolved/executed paths, and meaningful output/artifact assertions. Explain which contract cases and release branches were exercised and which remain blocked. Reuse existing PR evidence when publication is requested; do not publish by default. Read back and verify any published body.

Generated assets normally live under artifacts\tmp\<Configuration>\testsuite\<random-id>. Setting Microsoft_Testing_TestInfrastructure_TempDirectory_Cleanup=0 retains asset directories for diagnostics, but AcceptanceFixture.AssemblyCleanup still removes its cache. Capture provenance before fixture cleanup if the cache is needed. Prefer finally cleanup and existing cancellation tokens/bounded command helpers. Terminate only owned PIDs, unregister only exact run-owned package identities, and restore modified environment/state. Never remove broad shared caches, registrations, or workspace roots.

The app-model pipeline runs -Mode LeakCheck and publishes binlogs/TRX/layout/ manifest/resolved-assets/log diagnostics even on failure. On a shared local machine, do not use -RemoveStaleTestPackages without verifying ownership: that pipeline switch removes reserved-prefix packages for the current user. Preserve failures and evidence first; successful cleanup must not overwrite the test failure.

Maintaining this skill

For changes confined to this directory, validate the executable gate with:

python -m unittest discover -s .github\skills\testfx-acceptance-validation\scripts -p "test_*.py"

Check referenced repository paths, command flags, and CI/helper behavior against the current branch. These self-tests validate the evidence gate, not the product. Do not claim a composed-product or CI run from documentation/script checks.