Testing Strategy

Testing Strategy

Testing is an evidence-selection problem, not a contest to run the largest suite. Choose the smallest set that can falsify the changed behavior, then add one higher-level check only when it covers a boundary the lower level cannot. Keep execution environments reusable, but separate their evidence profiles: staging-smoke, security-proof, release-attestation, and nightly-stress. Use harness-feedback when a gate is reported as overloaded or misplaced.

Workflow

1. Freeze the acceptance criteria as observable outcomes.
2. Inspect the changed files and classify the risk.
3. Select the lowest useful test level from the matrix below and name the profile.
4. Run the fast gate first. If it fails, fix the cause before adding more tests.
5. Add a focused regression test for a confirmed bug or a changed invariant.
6. Test real boundaries only when the change crosses them.
7. Keep security-proof and release-attestation checks out of staging-smoke unless the acceptance criteria explicitly require that evidence.
8. For high-risk or long-horizon work, use a fresh-context verifier and store the command, revision, result, and skipped checks in a durable artifact.
9. When a verified stage becomes the input to another stage, seal that boundary with commit/tree, contract, input/output digests, and a fresh verdict. Mark an unavailable external prerequisite as BLOCKED; do not rerun unrelated accepted code merely because the following environment is unavailable.

Compact Matrix

The Stop hook runs the project's fast/default suite only when the working tree contains code or test changes. Projects with a complex suite may declare .claude/test-policy.json:

{
  "fast": ["python", "-m", "pytest", "-q", "tests/unit"],
  "integration": ["python", "-m", "pytest", "-q", "tests/integration"],
  "release": ["python", "-m", "pytest", "-q"]
}

fast is the automatic Stop gate. integration is additionally selected for high-risk changes when present. release is explicit or CI-only; do not make every edit pay the release-suite cost. Optional profiles make the separation explicit; a staging profile must not contain release-signing requirements.

Test Kinds

• Unit: isolated behavior and invariants; fast and numerous.
• Focused regression: a minimal test that was red before a fix and green after it. Keep it when it protects a real contract.
• Integration: one real boundary such as a database, filesystem, queue, or external adapter. Use a local/test dependency, never production.
• Contract: provider/consumer schema and serialization expectations.
• Smoke/E2E: a small number of real user or release paths; keep them stable.
• Property/fuzz: invariants over generated inputs; use for parsers, normalizers, state machines, and edge-heavy algorithms.
• Performance/security: only when the change or release claim needs it; preserve a fixed workload and baseline.
• Agent eval: test task completion, tool selection, recovery, and safety on a versioned golden set. Deterministic assertions come first; an LLM judge is an additional signal, never the sole proof of code correctness.

Agent Evidence Contract

An agent must report: revision, changed scope, commands actually run, exit status, relevant counts, environment constraints, and checks not run with a reason. “Tests passed” without command output or a durable evidence file is not proof. A generated test is a candidate until it reproduces the failure or asserts a stable contract; do not add broad snapshot tests merely to inflate coverage.

For a confirmed bug use bug-reproducer: reproduce first, then fix, then run the same test again. For a large/high-risk change use proof-verify: a fresh context must produce the final verdict. For a safe structural refactor use refactoring-safely and characterization tests before the transformation.