QA Strategy

QA Strategy

This skill is the QA route's test-system design discipline.

Core: A test suite's value is not how many tests it has. It is how much you would believe a green run.

Tier selection

The three-layer model in cc10x:building still governs — unit proves local behavior, integration proves boundary wiring, E2E proves system truth. QA owns the outer two, plus UI.

Choose the cheapest tier that can still fail for the right reason. An E2E test that would also fail for ten unrelated reasons is a bad detector: it fires often and tells you little. Push a check down a tier whenever the lower tier can catch the same defect.

But do not push everything down. The defects that survive good unit coverage are exactly the ones that live between components — serialization mismatches, transaction boundaries, retry storms, partial failures. Those only appear at the tier where the components are real.

The scenario matrix

Every feature gets scenarios in three classes. A plan with only the first is not a test plan, it is a demo.

Error-handling scenarios are where the bugs are

For each boundary the flow crosses, ask: what happens when it is slow, down, returns garbage, returns a partial result, or succeeds after the caller gave up? Each answer that matters is a scenario.

The most valuable single question: when this fails halfway, what state is left behind? Partial-failure state is where data corruption lives, and it is almost never covered by accident.

Input-space coverage — enumerate exhaustively, then reduce deliberately

The goal is to cover as many user actions and chains of actions as possible: every option in a dropdown, every date class, every filter, every combination that a real user could produce.

Step 1 — enumerate the full input space (do this exhaustively)

For every interactive surface in the flow, list every option a user can actually pick:

• every dropdown / select — every option, not "a representative one"
• every filter, toggle, checkbox, radio
• every date field — today, past, future, boundary of range, invalid, empty
• every free-text field — its equivalence classes
• every navigation path that reaches the same screen
• every chain — the ordered sequences of actions a user strings together

Discover, do not assume. Options are usually data-driven; reading the component tells you a <select> exists, not what is in it at runtime. Drive the real UI to enumerate actual values (see UI tier tooling below). An enumeration built from the code alone will miss the options that only appear for certain roles, tenants, or feature flags.

Record the full enumeration in the test plan even when you will not execute all of it. The enumeration is the coverage claim. What you skip must be visible.

Step 2 — reduce with a technique, not by taste

Full combinatorial coverage is not achievable and pretending otherwise produces an unbuildable plan: 5 filters × 4 options each is 1,024 combinations; add a date and a sort and you are past 10,000. Nobody runs that suite, so in practice it never gets written, and you end up with less coverage than an honest reduction would have given.

Reduce with a named technique, and record which one you used and what it leaves uncovered:

Default recipe: every-option-once for single controls, pairwise across combinations, boundary values on every range, plus full combinatorial on any combination that is known to be risky (billing, permissions, anything where two settings interact by design).

Pairwise is the highest-leverage tool here. Most combination defects come from two settings interacting, not five. All-pairs buys the large majority of that value for a small fraction of the runs.

Step 3 — chains, not just single actions

A user action rarely stands alone. Cover the sequences too:

• the primary chain — the full path a real user walks, end to end
• back / re-entry — leave mid-flow and return; is state preserved or correctly discarded?
• re-application — apply a filter, change it, clear it, apply another
• order variation — does filter-then-search behave like search-then-filter?
• repetition — the same action twice; the same submit double-clicked

Chains are where state bugs live, and they are almost never found by testing actions in isolation.

Feature flags are a coverage dimension, not a setup detail

A feature shipped behind a flag has two live code paths in production, and only one of them is new. Both need coverage:

The OFF path is where the expensive incidents live. A flag gets rolled back precisely when something is already going wrong, and that is the worst possible moment to discover the rollback does not restore a working system. If a flag is genuinely one-way, that is a finding to surface, not a row to skip.

Where entitlements or plan tiers exist alongside flags, they are independent gates. flag-on + entitlement-off must degrade correctly rather than 500 — a combination almost no plan covers, because each gate is usually tested alone.

Confirm a flag applied, do not confirm it was set. Asserting that the config call returned 200 proves the call returned 200. Caches, restart requirements, and per-pod config all break the leap from "set" to "in effect", and they break it silently.

Edge cases worth a default look

• boundaries: 0, 1, max, max+1
• concurrency: two of the same request at once; the same idempotency key twice
• ordering: out-of-order and duplicate messages
• time: expiry, timezone, clock skew, retry after timeout
• identity: unauthorized, wrong tenant, expired token
• data: unicode, very long strings, null vs. absent

Observation points — the pipeline-wide assertion

A scenario that asserts only the final response tests the response. It does not test the pipeline.

For each scenario, name what should be observable at every stage:

Why logs are a first-class assertion

Logs are the only cheap way to prove that a middle stage executed. A 200 at the edge is consistent with a worker that never ran, a retry that silently swallowed, or a branch that fell through.

Asserting logs also makes the suite an enforcement point for the project's logging standard: if a scenario asserts a structured log line with named fields and the line is missing, unstructured, or logged at the wrong level, that is a real finding about observability — found by a test, before an incident needs it.

Assert on level + message + structured fields, not on a substring of a formatted line. Substring assertions on log text are among the most brittle tests it is possible to write.

Log access strategy. How the harness reads logs differs sharply by environment (local stdout, container logs, a log platform); state the access method in the plan for the environment in use.

Test environment

Isolation model

Ranked by strength:

1. Ephemeral everything — fresh DB, fresh services per run. Strongest; slowest.
2. Fresh data, shared services — services stay up, schema/data reset between runs. Usually the right trade.
3. Namespaced in a shared environment — every run scopes itself by a unique prefix/tenant. Only when 1 and 2 are impossible; leaks are hard to detect and the failure mode is cross-run interference that looks like flake.

Never test against an environment someone else is using. A suite that intermittently fails because a colleague was clicking around teaches the team to ignore red.

Readiness, not sleeping

sleep 30 is a race condition with a comment. Gate on a real signal — health endpoint 200, port accepting, migration complete, topic created — with a bounded timeout and a loud failure.

An under-gated environment produces failures that look like product bugs. That is precisely how a team learns to distrust its own suite.

Teardown

Teardown must verify itself. "Ran docker compose down" is not evidence; "docker ps shows nothing from this run" is. A suite that leaks will eventually make its own machine unable to run it.

Determining the topology — ask, then detect, then confirm on screen

Never silently infer how to run the system. The order is fixed:

1. Ask the user first. They know whether there is a compose file that works, a staging environment, or a cloud-only deploy. One question here saves an hour of wrong inference.
2. Then detect, using whatever the machine actually offers — compose/Tilt/Skaffold files, testcontainers in the manifest, kind/k3d, available MCP servers, project skills, plugins, and agents that already know how to run this system.
3. Then present the findings on screen and get explicit confirmation before building anything. Show the proposed topology in the conversation — not only written to a file. A plan the user never actually looked at is not an approved plan, and the environment is the single most expensive thing to get wrong.

Write the confirmed topology to env-plan.md after the user confirms it, not before.