Atdd Mutate

Mutation Testing

Add a third validation layer to the ATDD two-stream testing approach. Acceptance tests verify WHAT, unit tests verify HOW, mutation testing verifies that the tests actually catch bugs.

Core Concept

Mutation testing introduces deliberate bugs (mutants) into source code, then runs the test suite. If tests fail, the mutant is killed (good). If tests pass despite the bug, the mutant survives (test gap found).

Source code → introduce mutation → run tests
                                     ├── tests FAIL → mutant killed ✓
                                     └── tests PASS → mutant survived ✗

A project with 100% code coverage can still have a 60% mutation score — meaning 40% of introduced bugs go undetected by the test suite.

When to Use

Run mutation testing after both test streams are green:

1. Acceptance tests pass (WHAT is correct)
2. Unit tests pass (HOW is correct)
3. Mutation testing — verify tests actually detect regressions

This is Phase 6 in the team-based ATDD workflow, or a standalone quality check at any point during development.

Approach: Custom Mutation Tool (Preferred)

The preferred approach is to build a custom mutation tool for the project. This follows the methodology Uncle Bob developed for empire-2025 — a project-specific tool that walks the AST/source tree, applies one mutation at a time, runs targeted tests, and reports survivors.

Why Custom is Preferred

• Tight integration with the project's test runner and source structure
• Targeted execution — run only the tests affected by each mutation
• Language-agnostic — works for any language, including those without established mutation frameworks
• No external dependencies — the tool lives in the project
• AST-level precision — understands the language's constructs natively

Architecture (4 modules)

1. Mutations — rules table (e.g., + → -, true → false, >= → >) plus matching logic that walks the AST/form tree
2. Runner — source-to-test mapping, test execution, pass/fail capture
3. Core — orchestration: read source → discover sites → apply one at a time → run tests → restore original → report
4. Hashing/Selection — hash each function (AST-level) and its covering tests; call dae_mutmap.py to mutate only changed functions and to update the manifest after the run. See the Differential Mutation Testing section.

Core Mutation Categories

For the full architecture and detailed reference, see references/frameworks.md.

Alternative: Existing Frameworks

When speed of setup is more important than tight integration, use an established mutation framework as a secondary option:

For install commands, configuration, and CLI reference, see references/frameworks.md.

Differential Mutation Testing

Mutation testing is slow — re-running it after a small change re-mutates every function. Differential mutation testing re-mutates only the functions whose code or covering tests changed, reusing cached results for the rest.

• Custom-tool path — the Hashing/Selection module hashes each function and its covering tests and calls dae_mutmap.py (select before the run, update after). Results live in a committed mutation-manifest.json beside the tool, so the saving reaches CI and every clone. A function is re-mutated when its code, its covering tests, or the mutation operator set changed. See ${CLAUDE_PLUGIN_ROOT}/references/differential-mutation.md.
• Framework path — Stryker (--incremental), PIT (withHistory), and mutmut have native incremental modes; enable the framework's incremental flag and commit its history file. Do not build a separate manifest for the framework path.

Workflow

Before Step 1, create one TodoWrite todo per step of this workflow (Steps 1–6), all at once — the full list up front, as a roadmap. Flip each todo to in_progress / completed as you go. See ${CLAUDE_PLUGIN_ROOT}/references/progress-indicator.md.

Step 1: Verify Prerequisites

Before running mutation testing, confirm:

• Both test streams are green (acceptance + unit)
• The project has meaningful unit tests (mutation testing runs against unit tests)
• No uncommitted changes (mutations modify source files temporarily)

Introversion pre-pass (recommended). If the engineer plugin is installed, run its dae_introvert.py <root> first — a cheap static scan that flags tests which can pass without asserting on SUT output (no assertion, or every assertion nested in a conditional/loop/try). These are guaranteed-weak tests; fixing them before mutation makes the run cheaper and the suite stronger, and the conditional-assertion class is caught even when no mutant happens to exercise the bypassed path. A configured manifest.introversion.backend (e.g. a deintroverter that backward-slices each assertion to the SUT) does the deeper data-flow classification; absent one, a built-in Python-AST fallback covers the cheap cases. A test flagged introverted and carrying a surviving mutant is a high-confidence vacuous test — prioritise it in Step 4.