Test Gap Analysis

Test Gap Analysis

Answer one question: which caller-visible production behaviors could change without an existing test failing? Mutation reasoning is a probe, not the goal. Inventory public outcomes first, then verify only credible gaps.

Decision flow

1. Set scope

Discover production and test files from manifests and file types. After a narrow search misses, inspect the current directory broadly before asking for paths.

When the request names a risk, turn it into a one-line public-outcome allowlist before reading code. An outcome is not in scope merely because the same method writes it. For money math, allow computed or returned amounts, rates, tier/boundary choice, percentage base/order, floors/caps, and rounding; exclude non-monetary state predicates (including derived booleans), identity, and formatting. Private code is in scope only to trace an allowed outcome.

Do not expand a focused request into a repository audit, plan artifact, or dashboard. Use source and tests directly for familiar frameworks. Invoke test-analysis-extensions only when discovery or assertion semantics are unclear.

2. Establish one baseline

Run the narrowest existing test command once. Choose it from the project manifest; Microsoft.Testing.Platform executables may require dotnet run. Confirm tests executed: exit 0 with build-only output is not green. If that one attempt cannot run the suite, do not troubleshoot the runner or try alternate commands for an advisory review; continue statically and label all candidates unverified. Do not infer a project-configuration cause from missing output; name a cause only when the command reports it.

Missing runner output limits only claims of empirical mutation survival. It does not make source-proven facts tentative: a public outcome with no reaching test is still No coverage, and an exact expected value derived from the unmodified implementation is still actionable. State the baseline limitation once, then give the static source/assertion conclusion directly instead of hedging every row.

For an advisory review such as "would tests catch this?", stop execution after that baseline. Source-to-assertion mapping is sufficient evidence for No coverage and Candidate survivor (unverified). Trace or run the unmodified code once only when an original value is unclear. Apply mutations only for explicit verification, an exhaustive audit, or closing gaps with tests.

Any focused mutation budget limits execution, not discovery. Keep every distinct unasserted public outcome in the inventory.

3. Inventory public outcomes

For each public entry point, map:

• input partitions: classifier arms, compound conditions, invalid and nearest-valid guard boundaries, and default cases;
• each independent observation: returned field/variant, exception type, invalid-input acceptance, public state transition, or external side effect;
• private-helper composition, constants/rates, rounding, retries, cancellation, and error propagation as observed through the public caller.

Use public input/sequence -> expected outcome -> existing assertion -> gap. One asserted return field does not cover another. One allowed result does not cover its denial.

Money math: inventory the no-op path, every rate/tier and exact boundary, operation order, percentage base or composition, floor/cap, and rounding. Trace private helpers through the public result. A test asserting only a broad range does not pin any exact amount. For each actionable money row, derive one witness input and its exact original result through the complete call chain; do not recommend a generic "assert the exact amount" without supplying that amount.

Ordered guards and retries: inventory invalid below minimum | first valid | last allowed or retryable | first blocked | later blocked. For an upper guard such as value >= limit, use limit - 1, limit, and limit + 1; the last witness exposes narrowing to value == limit. Inventory every accepted and rejected error class. When type matching is polymorphic, include a representative derived accepted type that would expose exact-runtime-type narrowing. A test at the first blocked value does not protect the last allowed or later blocked value.

Authorization: enumerate each relevant identity/role, resource class, and action from the caller's view. Untested false, forbidden, and unchanged-role outcomes are first-class security gaps. Do not analyze variants of an allowed path while a denial outcome remains uninventoried. Check each public surface:

• permission-returning APIs: every distinct role/resource class and every returned capability independently;
• action-dispatch APIs: each read/write/delete-style action branch, especially paths that must return denial;
• role/state transitions: accepted, rejected, invalid, null, and empty inputs, including outcomes that must leave state unchanged.

Reserve execution for wholly untested public branches before another variant of a partially covered helper. If more than five high-risk behaviors are unasserted, execute the top 3-5 and keep the rest visible as No coverage or Candidate survivor (unverified).

Execution never replaces the ledger. Before mutating or answering, classify every required outcome, including each invalid input, guard boundary, classifier arm, action, and denial.

Completeness checkpoint: before selecting findings, explicitly account for every independent mode/flag, both zero and negative for a <= 0 guard, every accepted exception class, and a representative derived accepted exception when matching is polymorphic. For a removed guard, trace the fallthrough: if it still produces the same public exception type, it is equivalent unless finer exception metadata is an established contract.

4. Admit only observable candidates

First replay each exact mutation against every existing asserted input or sequence with all arguments fixed. Any changed return, exception, state, or side effect is Likely killed; a dedicated single-purpose test is unnecessary. Never compare the mutant on one input with the original on another.

For survivors, choose a witness before execution or reporting and state witness -> original observation -> mutant observation. Reuse it in the smallest test. Admit it only when the last two differ publicly after tracing the full call chain; otherwise choose a distinguishing witness or drop it.

Exclude:

• edits that require inserting or reordering statements rather than changing or removing an existing expression, condition, constant, return, or side effect;
• edits that do not compile, including removal of a declaration whose value is still referenced;
• overflow behavior, exception message/ParamName metadata, or other semantics not established by the current contract, source intent, or tests;
• a removed guard or short-circuit that falls through to the same result, exception, state, and side effects;
• private representation changes that every public input sequence observes identically, even if the suite stays green;
• a mutation whose proposed test passes against both original and mutant;
• boundary edits that return the same value on the distinguishing input; for example, changing result < floor ? floor : result to <= is equivalent at equality because both branches return floor;
• a standalone auto-property or trivial one-line wrapper/predicate with no meaningful branch, calculation, or side effect, unless the user names it;
• hypothetical future impact, generated code, logging/formatting-only changes, impossible values, and duplicate syntax variants.

Missing direct assertions do not prove No coverage: first trace existing assertions through public callers and shared branches. Missing assertions make an observable candidate a survivor; they do not make an inert mutation meaningful.

5. Rank and classify

Rank: (1) security denials, financial outcomes, errors, and state changes; (2) wholly unasserted public outcomes; (3) boundaries or exact values reached by weak assertions; (4) alternate variants of already-asserted behavior.

Finish the inventory before selecting mutations or a verdict. One killed attempt, exception type, or switch arm does not clear its siblings.

Choose the verdict from the completed inventory:

• Strong when core branches and primary boundaries are protected and only a few validation or default-case variants remain;
• Mixed when meaningful coverage exists but at least one important outcome partition is unprotected;
• Weak when important outcomes are broadly unprotected.

A handful of validation gaps does not make an otherwise broad suite Mixed unless validation is the named risk or the gaps threaten security, data, or other contract-critical behavior.

When the inventory meets the Strong criteria above, lead with Strong and name the protected boundaries and dual assertions before listing minor gaps. Do not open with Mixed, "only core paths", or a risk-heavy dashboard.

Stop when existing assertions kill the remaining candidates or no credible public survivor remains. Do not mutate every operator merely to fill a report or calculate a score.

Outside explicit verification, an exhaustive audit, or a requested test addition, execute no mutations. Do not mutate to confirm obvious no coverage. For explicit verification, execute one representative candidate per distinct high-risk outcome in scope; do not stop after the first one or two while another guard, action branch, error class, or denial remains unclassified. Omit equivalent syntax variants.