Mutation Test

Bounded mutmut fallback

When a native extension is found, the supported mutmut fallback is mutate_only_covered_lines = false, but only when the JSON report says fallback.allowed: true. The fallback gate requires all of:

• one or more explicit --mutation-target values matching only_mutate;
• an observed or tool-generated --estimated-mutants count (do not guess);
• the intended --max-children value;
• an explicit --runtime-budget-seconds value; and
• the conservative estimate baseline_test_seconds × estimated_mutants ÷ max_children × 1.5 to fit the budget.

If any bound is missing or the estimate exceeds budget, shrink the mutation target or use a project-approved mutation harness whose coverage and stats phases start in fresh subprocesses. Never silently widen only_mutate to compensate for disabling covered-line selection.

4. Run Mutation Analysis

# JavaScript
npx stryker run

# Python, only after the preflight decision is recorded
python -m mutmut run --max-children 4

# Java
mvn org.pitest:pitest-maven:mutationCoverage

5. Classify and Report Results

Keep harness/tool failures, direct project-test failures, and mutant outcomes in separate fields. For an existing mutmut log, the preflight script can classify known native reload signatures:

python scripts/native_extension_preflight.py \
  --classify-mutmut-log mutation-run.log \
  --format json

Running stats combined with cannot load module more than once per process, module functions cannot set METH_CLASS or METH_STATIC, or a fatal segmentation fault whose report lists extension modules is harness_tool_failure_native_extension_reload. It must set counts_as_mutant_outcome: false; do not compute a mutation score from that run. failed to collect stats is supporting context, not a sufficient signature by itself. Only a direct test run that fails independently of the mutation harness is a project_test_failure. Killed, survived, timeout, and no-tests results are mutant outcomes only after preflight, baseline tests, stats collection, and the mutation harness complete successfully.

def parse_mutation_results(report_path):
    """Parse mutation testing report"""
    return {
        "execution_mode": "mutmut-covered-lines-preflight-safe",
        "harness_status": "passed",
        "project_test_status": "passed",
        "total_mutants": 150,
        "killed": 120,
        "survived": 25,
        "timeout": 5,
        "mutation_score": 80.0,
        "survivors": [
            {
                "file": "src/auth/validate.ts",
                "line": 45,
                "mutator": "RelationalOperator",
                "original": "age >= 18",
                "mutant": "age > 18",
                "status": "survived"
            }
            # ... more survivors
        ]
    }

Output Format

## Mutation Testing Report

**Module**: src/auth/
**Test Suite**: test/auth/
**Execution mode**: mutmut covered-line mode (native-extension preflight safe)
**Harness status**: passed
**Project baseline tests**: passed

### Summary

| Metric | Value |
|--------|-------|
| Total Mutants | 150 |
| Killed | 120 (80%) |
| Survived | 25 (17%) |
| Timeout | 5 (3%) |
| **Mutation Score** | **80%** |

### Status: PASSED (threshold: 80%)

### Survived Mutants (Highest Priority)

#### 1. `src/auth/validate.ts:45`
```diff
- if (age >= 18) { return "adult"; }
+ if (age > 18) { return "adult"; }

Problem: Boundary condition not tested Fix: Add test case for age = 18

2. src/auth/login.ts:23

- if (attempts < maxAttempts) { allow(); }
+ if (attempts <= maxAttempts) { allow(); }

Problem: Off-by-one boundary not tested Fix: Add test for attempts = maxAttempts

Recommended Test Improvements

1. Add boundary tests for validate.ts (3 survivors)
2. Add error path tests for login.ts (2 survivors)
3. Test null/undefined cases in session.ts (1 survivor)

Coverage vs Mutation Score

High coverage with low mutation score indicates weak assertions

## Integration with CI

### GitHub Actions Integration

```yaml
- name: Run mutation testing
  run: npx stryker run --reporters json

- name: Check mutation threshold
  run: |
    SCORE=$(jq '.metrics.mutationScore' reports/mutation/stryker-incremental.json)
    if (( $(echo "$SCORE < 80" | bc -l) )); then
      echo "::error::Mutation score $SCORE% below 80% threshold"
      exit 1
    fi

Optimization Tips

Incremental Mutation Testing

Only test changed code:

# Stryker incremental
npx stryker run --incremental

# PITest history
mvn pitest:mutationCoverage -DwithHistory

Target Critical Modules First

{
  "mutate": [
    "src/auth/**/*.ts",
    "src/payment/**/*.ts",
    "src/validation/**/*.ts"
  ]
}

Related Skills

• tdd-enforce - Enforce test-first development
• flaky-detect - Identify unreliable tests
• test-sync - Maintain test-code alignment

Script Reference

mutation_runner.py

Run mutation testing for project:

python scripts/mutation_runner.py --module src/auth

mutation_analyzer.py

Analyze and prioritize survivors:

python scripts/mutation_analyzer.py --report stryker-report.json

native_extension_preflight.py

Run the isolated Python import/test preflight and emit machine-readable evidence:

python scripts/native_extension_preflight.py \
  --test-selection tests/test_critical_contract.py \
  --mutation-target src/package/critical_contract.py \
  --estimated-mutants 120 --max-children 4 \
  --runtime-budget-seconds 900 --format json

References

• @$AIWG_ROOT/agentic/code/addons/testing-quality/README.md — Testing quality addon overview
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC framework context for quality gates
• @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/vague-discretion.md — Measurable quality thresholds and gate criteria
• @$AIWG_ROOT/docs/cli-reference.md — CLI reference