QA Tester

QA Tester

Core philosophy: one test runner, minimum tools, executable code is the spec. The fewer different tools a codebase has, the lower the cognitive load on every engineer and AI that touches it. Prefer in-box Microsoft APIs; every extra dependency is a cost, not a feature.

1. First principles

Each principle has a why so you can judge edge cases instead of pattern-matching on the rule.

1. Minimum tool surface. Two tools that do the job beat five that do it elegantly. Why: every extra framework is cognitive debt owed by the next person who opens the repo.
2. In-box first. Prefer APIs that ship with .NET or Windows. Why: a test that runs with dotnet test on a clean checkout is the test that actually gets run.
3. Code-first. A test is the spec. Do not duplicate intent into Gherkin + bindings + helpers. Why: every layer of indirection is another place the next AI has to look.
4. Readable beats clever. A new engineer should understand a test in under 30 seconds.
5. Lightest abstraction that still reads well. Inline → helper → screen object → fixture. Why: page objects are a symptom of test duplication, not a solution.
6. Stable selectors. Roles, accessible names, AutomationId. Never CSS tied to layout internals. Why: a fragile selector is a test that will break for reasons unrelated to what it's checking. If a stable selector doesn't exist, fix the product first.
7. Deterministic waits. Use framework auto-waits or in-box event subscriptions. Why: Thread.Sleep hides real race conditions and taxes everyone else's suite time.
8. Test the lowest layer that gives confidence. Unit → API/integration → UI. Why: UI tests are slow, flaky, and tell you less than the integration test that already covers the same rule.
9. Isolation. No shared mutable state. Each test sets up and tears down its own world. Why: ordered tests and shared fixtures produce heisenbugs whose failure mode depends on run order — impossible to reproduce locally and demoralising for everyone on call.
10. Minimal change. Edit small. Don't rewrite suites unless asked. Why: a "while I'm in here" refactor turns a 5-line fix into a 500-line review and blocks the actual bug shipping.
11. Test project must be simpler than the app. One {Project}.Tests project by default. Mirror the product folder layout 1:1. Split into more projects only when a real constraint (headless CI, massive scale, different TFM) forces it. Why: a test project fractured into more pieces than the app it tests is absurd — every extra csproj duplicates infrastructure, slows the loop, and makes newcomers wonder which project to open.
12. Performance data is a byproduct of functional tests. If a code path already runs in CI under a unit / integration / UI test, attaching a lightweight in-box collector gives you perf coverage equal to your functional coverage at near-zero marginal cost. Why: a parallel "perf suite" that re-implements the same sign-in / order / search scenarios is pure duplication, and it creates coverage gaps wherever the two suites drift. Dedicated load/stress/soak tests exist only for what functional tests genuinely cannot express: concurrency, saturation, sustained throughput, and memory growth over time. See §6a.

2. The two-tool stack

That's the whole stack. System.Windows.Automation is in-box (no NuGet), so it doesn't appear in the package list but is the default for any Windows desktop test.

Performance collection (§6a) uses only in-box instrumentation — EventPipe / dotnet-trace, dotnet-gcdump, dotnet-counters, SQL Server Extended Events, Playwright tracing, ETW via wpr.exe — plus one Microsoft-owned load generator (Microsoft.Crank). Collectors are runtime/OS facilities, not "tools", so the two-tool rule is unchanged.

Quick exclusion list (see references/legacy-bdd.md for inherited-project support):

Before adding any third tool, write one sentence justifying it. If the sentence is "it would be more convenient", reject it.

3. Layer → test type (the whole decision matrix)

Default rule of thumb: MSTest + System.Windows.Automation covers every .NET desktop scenario; MSTest + Playwright covers every web scenario. If you reach for a third tool, re-read §2.

4. Writing style — code-first, business-readable second

When the user says:

Given a valid user, when they sign in, then they should see the dashboard

Write this:

// @under-test: src/Foo.Web/Pages/SignIn.razor
// @area: auth   @layer: ui-web
[TestClass]
public class SignInTests : PageTest
{
    [TestMethod]
    [TestCategory("auth"), TestCategory("smoke")]
    [Description("Valid user signs in and lands on dashboard")]
    public async Task Valid_user_signs_in_and_lands_on_dashboard()
    {
        // Given: a valid user on the sign-in page
        await Page.GotoAsync("/sign-in");

        // When: they sign in with valid credentials
        await Page.GetByLabel("Email").FillAsync("valid.user@example.com");
        await Page.GetByLabel("Password").FillAsync(Environment.GetEnvironmentVariable("TEST_USER_PASSWORD")!);
        await Page.GetByRole(AriaRole.Button, new() { Name = "Sign in" }).ClickAsync();

        // Then: the dashboard is visible
        await Expect(Page).ToHaveURLAsync(new Regex("/dashboard$"));
        await Expect(Page.GetByRole(AriaRole.Heading, new() { Name = "Dashboard" })).ToBeVisibleAsync();
    }
}

The Given/When/Then phrasing lives in test name + comments + [TestCategory] tags + [Description]. No .feature file, no bindings, no regex maintenance, no indirection. A reporter that lifts [Description] + [TestCategory] into Markdown covers every "but stakeholders need to read it" objection without introducing BDD tooling.

5. Discoverability — finding tests from a code path (and vice versa)

For Code ↔ UI mapping (web routes, desktop menu breadcrumbs), see the solution-patterns skill. This section covers Code ↔ Test only.

This section is load-bearing. Renaming any convention here breaks every future locate-or-create query.

5.1 Test project naming

Default: one test project per product project — {Project}.Tests, sibling to {Project}. Why: a test project more complex than the app it tests is absurd; the whole point of §1.1 (minimum tool surface) applies to the test tree too. One csproj, one namespace root, one dotnet test command.

Split into multiple projects only when one of these is actually true:

• Headless CI constraint. UI tests need a graphical session; unit and integration tests must pass on a headless build server. Split into {Project}.Tests (headless-safe) and {Project}.Tests.UI (requires a desktop session). Tag the UI csproj so CI can skip it in headless runs, e.g. <IsTestProject>true</IsTestProject> plus a [TestCategory("ui-interactive")] filter.
• Large codebase. Hundreds of product files and thousands of tests where a single csproj's compile + test-discovery time noticeably slows the loop. This is rare in practice.
• Fundamentally different runtime. e.g. product targets net8.0 but one test layer needs net48 for COM interop.
• Dedicated perf/load/soak suite. {Project}.Tests.Perf/ houses Microsoft.Crank scenarios and long-running soak tests. Its runtime constraints are genuinely different: long durations, non-parallel with the functional suite, its own CI job, statistical gating instead of pass/fail per run. See §6a.5. Do not put piggyback perf collectors here — those belong next to the functional tests whose coverage they inherit.

Do not split because "separation of concerns" or "unit vs integration is cleaner" — those are solved by folder structure and [TestCategory] tags, not by csproj boundaries. Extra csprojs bring: duplicated [AssemblyInitialize] code, duplicated TestSandbox/TestHelpers, multiple InternalsVisibleTo grants, separate Directory.Packages.props entries, and separate test hosts that all have to be kept in sync. Pay that cost only when a real constraint forces it.

Naming, whichever variant you pick: {Project}.Tests or {Project}.Tests.UI. One spelling forever. Never {Project}Tests, never Test.{Project}, never {Project}.UnitTests.

5.2 File-to-file mapping (deterministic)

Folders inside the test project mirror folders inside the product project 1:1. Never flatten — reversibility is the whole point. The product sub-path becomes the test sub-path verbatim, with Tests appended to the class-file name.

Default (single {Project}.Tests project):

Split form (only when §5.1's split criteria apply): the UI-requiring tests move to {Project}.Tests.UI/ using the same sub-path mirror; everything else stays in {Project}.Tests/. No other splits.

Namespace rule: every test file uses the single root namespace {Project}.Tests (or {Project}.Tests.UI), regardless of which subfolder it lives in. Why: sub-namespacing per folder forces a cross-file using hunt every time a test helper moves, with no discoverability benefit — the folder is right there in the solution explorer. The @under-test header (§5.3) is the authoritative coverage link, not the namespace.

Locating a test file: given {Project}/Foo/Bar/Baz.cs, the test is at {Project}.Tests/Foo/Bar/BazTests.cs. No lookup, no search — pure string transform. That is the whole point of the mirror.

5.3 Mandatory test-file header

Every test file declares the product file it covers:

// @under-test: src/Foo.Bar/Services/OrderService.cs
// @area: orders   @layer: unit   @ticket: JIRA-1234

Rules: @under-test is required, repo-relative, forward slashes, no globs. Multi-file tests list at most three paths comma-separated; more means the test is too broad — split it. @layer is one of unit | integration-api | integration-db | ui-web | ui-wpf | ui-maui | perf. Optional: @perf-capture: always forces piggyback collection even when QA_PERF_CAPTURE is unset — use only for known hot paths whose perf you always want a baseline for.

Why this matters: reverse coverage queries become one ripgrep (rg "@under-test:\s*src/Foo.Bar/Services/OrderService.cs" Tests/) instead of a semantic search.

5.4 Locate-or-create algorithm

Adding/modifying a test for product file P:

1. Compute expected path T from §5.2.
2. If T exists → edit it.
3. Else rg "@under-test:.*P" across Tests/ — if a hit exists at a non-canonical path, edit it and leave // TODO: relocate to {T}.
4. Else search by symbol name ({Name}Tests). Same rule.
5. Else create T with the @under-test header.

5.5 Coverage verification + impact-scoped test selection

scripts/test_map.py automates the §5.2 mirror check and the §5.3 @under-test header lookup. Zero required arguments — auto-discovers the {Project} + {Project}.Tests pair from the current directory. Cross-platform (Windows, Linux, macOS — runs in containers, CI, Codex).

# Coverage gap report (default — run from repo root, no arguments needed)
python test_map.py
#   Output: table of OK / MISSING / ORPHAN / WRONG_HEADER per product file

# Impact analysis: which tests cover my recent changes?
python test_map.py --affected HEAD~1
#   Output: direct changes + transitive dependents (2-level type-name grep) + filter string

# JSON for AI consumption
python test_map.py --affected HEAD~3 --json

The --affected mode produces a dotnet test --filter string that includes every mirror-path test + every crosscutting test whose @under-test header mentions any affected file + TestCategory=critical. The AI supplements this list by using whatever reference-finding capability its IDE/platform provides — e.g. find_references in VS Code (Claude Code, GitHub Copilot), find_usages in JetBrains (Roo Code), or Go to References in Visual Studio — to chase shared interfaces/base classes where string-grep may miss semantic dependents. See references/test-selection.md for the detailed workflow.

Standard severity categories for filtering (§7 naming convention applies):

What to tag critical — a test earns this tag when ALL of these are true:

1. Fast. Runs in under 1 second. If it needs ProjFS, a database, a browser, an exe launch, or a network call, it is not critical — it belongs in a slower category. The whole point of critical is a 5–10 second feedback loop.
2. Covers a code path where a bug causes data loss, crashes, or silent corruption. Examples: config load/save (losing projections), manifest read/write (losing the file list), hash comparison (false "in sync"), single-instance mutex (app refuses to start), provider state machine (delete propagates to Source instead of just manifest).
3. Is a unit or service test, not an integration or UI test. critical tests must run on any machine — headless CI, a locked-down VM, a container, a developer laptop — with zero external dependencies. If it needs the Client-ProjFS feature, a desktop session, admin elevation, or a specific folder on disk, tag it with the appropriate category instead.

Why this matters: dotnet test --filter "TestCategory=critical" is the developer's 10-second sanity check between edits. If critical tests include slow ProjFS integration tests, the filter becomes useless and developers stop using it. If critical tests miss the config-persistence path, a developer ships a bug that loses every user's projection list. The tag is the contract between "this is cheap enough to run constantly" and "this catches the bugs that matter most."

dotnet test --filter "TestCategory=critical"                                                          # fast dev loop
dotnet test --filter "TestCategory!=stress&TestCategory!=ui-interactive&TestCategory!=requires-elevation"  # PR / CI (headless, no admin)
dotnet test --filter "TestCategory!=requires-elevation"                                               # developer machine (has desktop, no admin)
dotnet test                                                                                           # release gate (full, interactive, admin)

6. Examples

Four examples, one per major layer. Each is runnable; no pseudo-code.

6.1 Unit test

// @under-test: src/Foo.Bar/Services/OrderTotals.cs
// @area: orders   @layer: unit
[TestClass]
public class OrderTotalsTests
{
    [TestMethod]
    public void Total_sums_line_items_with_tax()
    {
        var items = new[] { new LineItem(10m), new LineItem(20m) };
        Assert.AreEqual(33m, new OrderTotals(taxRate: 0.10m).Compute(items));
    }
}

6.2 API integration — replaces most "page renders" UI tests

WebApplicationFactory<Program> needs the product's Program.cs to expose Program as a type. In a top-level-statements project add this at the bottom of Program.cs:

public partial class Program { }

Then the test (needs using System.Net.Http.Headers;, using System.Text;, using Microsoft.AspNetCore.Mvc.Testing;):

// @under-test: src/Foo.Web/Controllers/OrdersController.cs
// @area: orders   @layer: integration-api
[TestClass]
public class OrdersApiTests
{
    private static WebApplicationFactory<Program> _factory = null!;
    private HttpClient _client = null!;

    [ClassInitialize]
    public static void Setup(TestContext _) => _factory = new WebApplicationFactory<Program>();

    [ClassCleanup]
    public static void Teardown() => _factory.Dispose();

    [TestInitialize]
    public void SetupClient() => _client = _factory.CreateClient();

    [TestMethod, TestCategory("orders"), TestCategory("smoke")]
    public async Task Get_orders_returns_empty_array_for_new_user()
    {
        var creds = Convert.ToBase64String(
            Encoding.UTF8.GetBytes("new.user@example.com:" + Environment.GetEnvironmentVariable("TEST_USER_PASSWORD")));
        _client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", creds);

        var response = await _client.GetAsync("/api/orders");
        response.EnsureSuccessStatusCode();
        Assert.AreEqual("[]", await response.Content.ReadAsStringAsync());
    }
}

Any test that was reaching for Playwright just to assert "page loads, contains X" belongs here instead. 10× faster, no browser dependency.

6.3 Playwright web smoke — same MSTest runner

// @under-test: src/Foo.Web/Pages/SignIn.razor
// @area: auth   @layer: ui-web
[TestClass]
public class SignInSmokeTests : PageTest
{
    [TestMethod, TestCategory("auth"), TestCategory("smoke")]
    public async Task Valid_user_signs_in_and_lands_on_dashboard()
    {
        await Page.GotoAsync("/sign-in");
        await Page.GetByLabel("Email").FillAsync("valid.user@example.com");
        await Page.GetByLabel("Password").FillAsync(Environment.GetEnvironmentVariable("TEST_USER_PASSWORD")!);
        await Page.GetByRole(AriaRole.Button, new() { Name = "Sign in" }).ClickAsync();
        await Expect(Page).ToHaveURLAsync(new Regex("/dashboard$"));
        await Expect(Page.GetByRole(AriaRole.Heading, new() { Name = "Dashboard" })).ToBeVisibleAsync();
    }
}

6.4 WPF / WinUI smoke via System.Windows.Automation (in-box)

Every interactive control gets AutomationProperties.AutomationId in the format Screen.Element. No AutomationId → fix the XAML first, don't write the test.

<Button      AutomationProperties.AutomationId="SignIn.Submit" Content="Sign in" />
<TextBox     AutomationProperties.AutomationId="SignIn.Email" />
<PasswordBox AutomationProperties.AutomationId="SignIn.Password" />

// @under-test: src/Foo.Desktop/Views/SignInView.xaml
// @area: auth   @layer: ui-wpf
using System.Windows.Automation;

[TestClass]
public class SignInSmokeTests
{
    private Process _proc = null!;
    private AutomationElement _window = null!;

    [TestInitialize]
    public void Launch()
    {
        _proc = Process.Start(@"bin\Release\MyApp.exe")!;
        _proc.WaitForInputIdle();
        _window = WaitFor(() => AutomationElement.FromHandle(_proc.MainWindowHandle), TimeSpan.FromSeconds(5));
    }

    [TestCleanup] public void Close() { if (!_proc.HasExited) _proc.Kill(); _proc.Dispose(); }

    [TestMethod, TestCategory("desktop"), TestCategory("smoke")]
    public void Valid_user_signs_in()
    {
        ById("SignIn.Email").SetValue("valid.user@example.com");
        ById("SignIn.Password").SetValue("***");
        ById("SignIn.Submit").Invoke();

        var heading = WaitFor(() => ById("Dashboard.Title"), TimeSpan.FromSeconds(5));
        Assert.AreEqual("Dashboard", heading.Name);
    }

    // Minimal intent-level wrappers so test bodies don't see UIA pattern ceremony.
    private Ui ById(string id) => new(_window.FindFirst(TreeScope.Descendants,
        new PropertyCondition(AutomationElement.AutomationIdProperty, id))
        ?? throw new InvalidOperationException($"AutomationId not found: {id}"));

    private readonly record struct Ui(AutomationElement Element)
    {
        public string Name => (string)Element.GetCurrentPropertyValue(AutomationElement.NameProperty);
        public void Invoke()           => ((InvokePattern)Element.GetCurrentPattern(InvokePattern.Pattern)).Invoke();
        public void SetValue(string t) => ((ValuePattern)Element.GetCurrentPattern(ValuePattern.Pattern)).SetValue(t);
    }

    // The ONLY place polling lives. Test bodies never see a sleep.
    private static T WaitFor<T>(Func<T?> probe, TimeSpan timeout) where T : class
    {
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            var r = probe();
            if (r != null) return r;
            Thread.Sleep(50);
        }
        throw new TimeoutException("UIA element not found within " + timeout);
    }
}

Zero NuGet beyond MSTest. Zero driver processes. Zero HTTP hops. using System.Windows.Automation; is the entire import. If the 50 ms poll ever proves insufficient, upgrade to Automation.AddAutomationEventHandler(...) (also in-box) for true event-based waits.

6a. Performance — piggyback by default, dedicated load suite only when needed

The 2-in-1 rule. Every functional test in §6 already exercises a code path end-to-end. When the env var QA_PERF_CAPTURE=1 is set (default in the perf CI job, off locally), the same test run also emits a perf/ folder of in-box telemetry. Functional coverage becomes performance coverage — if a path has a test, it has perf data. No parallel perf suite, no duplicated scenarios, no "we forgot to perf-test the refund flow" gaps.

Piggyback records, load suite gates. Per-test perf data is never asserted against absolute thresholds — that flakes on slow CI agents and encodes machine speed into your suite. Regressions are surfaced by diffing perf artifacts across runs (baseline vs. PR). Only the dedicated load suite ({Project}.Tests.Perf/) gates pass/fail, and it gates on statistical baselines across many samples — not single-run numbers.

6a.1 Per-layer collectors (all in-box)

EventPipe is the in-box .NET diagnostic pipe — Microsoft.Diagnostics.NETCore.Client drives it from test infrastructure so no external dotnet-trace process is required. ETW (wpr.exe) is the Windows equivalent for UI-thread and render work. None of this adds a NuGet dependency beyond Microsoft.Diagnostics.NETCore.Client.

6a.2 The artifact contract (identical across layers)

Every test — piggyback or dedicated — leaves behind the same folder shape so one parser in an AI agent handles all of them:

TestResults/{run}/{FullTestName}/perf/
├── manifest.json        # test name, @under-test, @layer, git sha, wall-clock start/stop, machine
├── trace.nettrace       # EventPipe: CPU samples + GC + allocs + contention
├── counters.json        # dotnet-counters delta across the test window
├── sql.xel              # integration-* only: XEvent capture for this test's window
├── ef-queries.ndjson    # integration-api only: one row per SQL, with duration + rows
├── otel.ndjson          # integration-api only: OTLP spans incl. EF Core spans
└── ui/                  # ui-* only
    ├── playwright-trace.zip
    ├── perf-entries.json
    └── network.har

The dedicated load suite ({Project}.Tests.Perf/) adds load/crank.json and the bracketing gcdump-*.gcdump files to the same shape. Why identical shape: "show me the regression" is the same query whether the data came from a 5 ms unit test or a 10-minute soak run — one diff algorithm, one code path in the AI agent.

6a.3 The piggyback hook (one file, reused by every test project)

// TestInfrastructure/PerfCapture.cs — lives in {Project}.Tests/TestInfrastructure/
// Opt-in via env var so the local TDD loop stays fast.
[TestClass]
public static class PerfCapture
{
    private static EventPipeSession? _session;
    public static bool Enabled => Environment.GetEnvironmentVariable("QA_PERF_CAPTURE") == "1";

    [AssemblyInitialize]
    public static void Start(TestContext ctx)
    {
        if (!Enabled) return;
        var client = new DiagnosticsClient(Process.GetCurrentProcess().Id);
        var providers = new[]
        {
            new EventPipeProvider("Microsoft-DotNETCore-SampleProfiler", EventLevel.Informational),
            new EventPipeProvider("Microsoft-Windows-DotNETRuntime", EventLevel.Verbose, 0x1 | 0x80 | 0x4000080018),
        };
        _session = client.StartEventPipeSession(providers, requestRundown: true);
        _ = Task.Run(() => _session.EventStream.CopyTo(File.Create(PerfPaths.TraceFile)));
    }

    [AssemblyCleanup]
    public static void Stop() { _session?.Stop(); _session?.Dispose(); }
}

That's the whole piggyback machinery for unit and integration-api. Layer-specific collectors (XEvent session, Playwright tracing, wpr) live in small siblings (SqlXEventCapture.cs, PlaywrightPerfCapture.cs, WprCapture.cs) and are activated by the same env var.

6a.4 Example — N+1 detection from an integration-api test

// @under-test: src/Foo.Web/Controllers/OrdersController.cs
// @area: orders   @layer: integration-api
[TestMethod, TestCategory("orders")]
public async Task Get_orders_returns_user_orders_with_customer_name()
{
    var response = await _client.GetAsync("/api/orders?userId=42");
    response.EnsureSuccessStatusCode();
    // Functional assertion — the only one that gates the test.
    Assert.IsTrue((await response.Content.ReadAsStringAsync()).Contains("Jane Doe"));
}

With QA_PERF_CAPTURE=1, this run also emits perf/ef-queries.ndjson. If the endpoint regressed from 1 SQL statement into a per-row SELECT Customer WHERE Id = @p loop, the file contains 51 rows instead of 1. The AI agent diffs the file against the baseline, spots the N+1 immediately, and maps it back via the @under-test header to OrdersController.cs. No perf assertion, no flake, no duplicated scenario.

6a.5 The dedicated {Project}.Tests.Perf/ suite

This is the fourth legitimate §5.1 split. It houses only scenarios that functional tests cannot express:

• Load: target RPS against WebApplicationFactory-hosted app or a deployed env. Tool: Microsoft.Crank (github.com/dotnet/crank — the benchmarking infrastructure used by the .NET team itself for the TechEmpower runs). Emits crank.json with p50/p95/p99, RPS, errors.
• Stress: push past saturation until the first error; record the break point.
• Soak: sustained load for N hours with bracketed dotnet-gcdump before/after. Diffing the two gcdumps surfaces retained types, counts, and GC roots — the "undisposed objects pinned to root" detector, built from in-box tools only.
• Micro-benchmarks for @perf-critical methods: BenchmarkDotNet jobs, runnable under the same MSTest runner via an [TestMethod] that shells out to BDN and asserts against its statistical baseline.

The suite gates on statistical baselines across samples (e.g. p95 latency must not regress by >10% with p<0.05), never on single-run numbers.