Ln 55 Test Suite Auditor

Test Suite Auditor

Goal: Audit the test portfolio as a read-only lifecycle and confidence system. Determine which important failures it detects, which evidence is untrustworthy or obsolete, and which additions, changes, consolidations, retirements, or explicit omissions produce the smallest sustainable portfolio.

Execution contract: The checklist defines completion. Track each item internally as PENDING, PROVEN with evidence, CLEARED with evidence its condition is absent, or UNPROVEN with a gap; reading, delegation, tool failure, a zero exit status, or a self-reported success is not proof; only the observed outcome is. Reconcile after each section. Before returning, resolve all PENDING, count only PROVEN and CLEARED, and apply verdict and approval rules to every gap. Preserve intent, scope, and existing authorization. Continue authorized work; ask only for consequential unresolved choices or required external approval. When no one can answer during the run, state the exact question and apply the skill's verdict for the remaining gap instead of waiting or guessing. Scale depth to material risk without skipping checks. Preserve dependency and safety order; otherwise choose an appropriate verification method. Accept equivalent user or repository evidence; no other skill, named artifact, or complete lifecycle is required. Preserve source requirement and decision IDs. Bind reused evidence to relevant source versions, dirty changes, configuration, and environment; invalidate only affected claims. On continuation, reconcile task, authorization, current state, and unresolved evidence. For long work, return a compact continuation record or update an already authorized artifact; read-only skills do not persist it. Distinguish artifact readiness, verified behavior, and external-action authority. Prepare authorized work before required approval. If blocked by an instruction, cite its exact source and unresolved boundary; do not invent approval gates from caution.

Tool Routing

Run only safe test and diagnostic commands. Do not rewrite snapshots, update golden files, regenerate fixtures, or accept changed output during the audit.

Evidence Rules

• Coverage indicates execution, not proof. Require an assertion or observable oracle for important behavior.
• A slow test is not low-value when it uniquely protects a critical journey; a fast test is not high-value when it proves framework behavior.
• A flaky failure must be separated from an intermittently failing product dependency or genuinely nondeterministic requirement.
• Deletion recommendations require proof that the test basis is obsolete or that other evidence covers every still-required behavior and failure mode with equal or better trust.
• Merge recommendations require demonstrated duplicate or fragmented coverage and must preserve distinct business and failure scenarios, oracle strength, and failure localization; a larger test is not inherently better.
• Known regression guards and the only proof of a rare critical edge case are not deletion candidates merely because a numeric value heuristic is low.
• A real dependency is not inherently a test defect. Judge whether its version, state, ownership, reset, availability, and failure evidence make the result reproducible.
• Keep portfolio action (KEEP, ADD, UPDATE, MERGE, DELETE, NO_TEST) separate from execution status (PASS, FAIL, BLOCKED, UNPROVEN, QUARANTINED). NO_TEST requires existing proof, another control, or explicit residual-risk acceptance.
• Do not require a new registry by default. Prefer traceability derivable from repository-native test paths, behavioral names, tags, requirements, CI configuration, and review evidence.
• External testing guidance becomes actionable only when it explains a concrete weakness in this suite.

Checklist

1. Map the Portfolio and Baseline

• Detect test runners, configurations, commands, directories, fixtures, factories, snapshots, golden files, manual scripts, coverage, and mutation tooling.
• Map source domains and critical entrypoints to unit, integration, contract, end-to-end, and manual test surfaces.
• Trace requirements, product risks, incidents, public contracts, and changed behavior to tests and results where evidence exists; identify tests with no current test basis and basis elements with no credible proof.
• Read repository instructions and CI configuration to identify required suites, environment assumptions, retries, sharding, and exclusions.
• Run representative suites and required gates where feasible; record environment, duration, exit status, discovered/executed scope, failures, skips, and retries. Flag required suites that silently select no tests.
• Separate generated, vendored, example, migration-history, and infrastructure fixtures from product tests before evaluating the portfolio.
• Keep the audit read-only and disclose any caches or test artifacts created by permitted commands.

2. Audit Product Value and Coverage

• Identify uniquely critical local logic: money, authentication, authorization, data integrity, algorithms, domain rules, destructive operations, and irreversible workflows.
• Trace each critical behavior to at least one test whose oracle would fail for the corresponding defect; name/path matches and line coverage are only discovery evidence.
• For tests of configuration, queries, schemas, adaptation, validation, or failure handling, trace the asserted behavior to a concrete business rule or outcome; technical coverage alone does not establish value.
• Check whether end-to-end tests cross the production-shaped boundaries relevant to the risk and prove the terminal durable or user-visible outcome, not only an intermediate status, page, or mock call.
• Test value and boundary: Require every test to detect a concrete defect in this product's business logic and name the protected business outcome. Prefer E2E through user or external-system boundaries; use integration or unit tests only for business scenarios difficult to exercise reliably through E2E. Reject platform, trivial-wiring, implementation-detail, and duplicate proof with no distinct business failure signal.
• Inspect error, retry, timeout, authorization, concurrency, migration, compatibility, and recovery behavior where those failures are plausible and costly.
• Use coverage data to locate unexecuted critical paths, then inspect behavior and assertions before reporting a gap.
• Classify every material gap and in-scope affected test as KEEP, ADD, UPDATE, MERGE, DELETE, or NO_TEST, justified by impact, plausible failure, uniqueness, trust, and maintenance cost; use UPDATE when valuable intent remains but its basis, setup, boundary, assertion, or oracle must change.

3. Audit Isolation and Determinism

• Check shared database, filesystem, environment, process, network, cache, clock, random generator, and global state for leakage between tests.
• Check setup and teardown on success and failure, unique test data, transaction boundaries, cleanup, and parallel-safe resource ownership.
• Diagnose suspected flakes using only discriminating modes: alone, in-suite, fixed-seed repetition, shuffled/reversed order, or parallel execution. Preserve the first failure, order, seed, worker, and environment; stop when evidence identifies the cause or the bounded diagnostic budget ends.
• Detect time, timezone, locale, randomness, sleep, scheduler, and race sensitivity; require controllable clocks or seeds where behavior depends on them.
• For real dependencies and emulators, verify version pinning, readiness, namespace/state reset, failure cleanup, credentials, and CI availability instead of assuming either real or mocked is preferable.
• Review retries and quarantine rules so they preserve the first failure and reproducibility data rather than converting an initial failure into a silent pass.
• Require every quarantine to remain visible outside the passing result, with an owner or decision path and a concrete recovery, replacement, or retirement trigger; quarantine is an execution state, not a portfolio action.
• Distinguish test flakiness from real intermittent product defects using repeated evidence, logs, and the failing path.