Mutation Testing

Mutation Testing (pitest)

Mutation testing for Kotlin/Java Gradle projects using pitest via the info.solidsoft.pitest plugin. Non-intrusive: mutates bytecode, requires no production-code changes.

When to use this skill

• User asks: "add mutation testing", "check mutation score", "why does this mutant survive", "strengthen tests", "is this class well-tested".
• A survivor appeared in a pitest report and needs killing.
• A project wants a CI gate on mutation score.

Prerequisites

• Gradle project (Kotlin DSL build.gradle.kts primary, Groovy build.gradle supported via analogous blocks).
• Tests already exist. If not, run /ralph-coverage or /kotest-create first — mutation testing on an empty suite only produces NO_COVERAGE noise.
• Clean or near-clean working tree.

Preflight — run before any pitest invocation

Pitest fails cryptically (silent UNKNOWN_ERROR from the coverage minion) when the project itself doesn't compile or when the test runtime has misaligned JUnit jars. Always run these checks before ./gradlew pitest so failures surface with their real cause, not pitest's.

# 1. Test sources must compile — pitest can't help if the project itself is broken.
./gradlew compileTestKotlin compileTestJava --quiet

# 2. Find the project's actual junit-platform-launcher version.
./gradlew dependencyInsight --configuration testRuntimeClasspath \
  --dependency org.junit.platform:junit-platform-launcher 2>/dev/null \
  | head -1

# 3. Check that Kotlin bytecode keeps line-number debug info (pitest needs it).
grep -RnE 'Xno-source-debug-extension|-g:none' build.gradle.kts settings.gradle.kts 2>/dev/null

If step 1 fails, abort and surface the compile error — do not attempt pitest. If step 2 reports a version, use it to pick junit5PluginVersion (see next section). If step 3 finds either flag set, pitest will produce bogus line numbers or silently fail on mutators that need source info — remove the flag or exclude those modules.

Capabilities

1. Bootstrap

Detect whether pitest is already configured:

grep -n "info.solidsoft.pitest" build.gradle.kts settings.gradle.kts **/build.gradle.kts 2>/dev/null

If not configured, propose this diff to the root build.gradle.kts:

plugins {
    id("info.solidsoft.pitest") version "1.19.0"
}

pitest {
    pitestVersion.set("1.19.0")
    targetClasses.set(setOf("<group.package>.*"))          // derive from rootProject.group
    targetTests.set(setOf("<group.package>.*"))
    mutators.set(setOf("STRONGER"))                        // do NOT use ALL — NPE- and equivalent-mutation-prone
    features.set(listOf(
        "+FLOGCALL",                                       // built-in: silence many logger-call mutations
        // `+fkotlin` is auto-enabled by the junit5 plugin when detected — filters Kotlin bytecode-junk mutations
    ))
    avoidCallsTo.set(setOf(
        // FLOGCALL and avoidCallsTo overlap partially but neither fully covers the other on real Kotlin+SLF4J code.
        // Verified empirically: removing these re-introduces 7+ logger-call survivors per scoped run.
        "kotlin.jvm.internal", "kotlin.Metadata",
        "org.slf4j.Logger",
        "org.apache.logging.log4j.Logger",
        "java.util.logging.Logger",
    ))
    excludedMethods.set(setOf(
        // Kotlin data-class synthesized members — generated, behavior guaranteed
        "component*", "copy", "hashCode", "equals", "toString",
    ))
    junit5PluginVersion.set("1.2.1")                       // JUnit 5 projects only — pin to project's Platform version
    outputFormats.set(setOf("HTML", "XML"))
    threads.set(Runtime.getRuntime().availableProcessors())
    timestampedReports.set(false)                          // stable path for parsing
    fullMutationMatrix.set(true)                           // names covering tests for SURVIVED mutants too — see §3
    exportLineCoverage.set(true)                           // emits coverage.xml alongside mutations.xml
    historyInputLocation.set(file("build/pitest/history.bin"))     // incremental analysis — read + write same file to skip re-running killed/survived mutants on unchanged classes
    historyOutputLocation.set(file("build/pitest/history.bin"))    // the gradle-pitest-plugin exposes explicit paths, not the `withHistory` CLI flag
    jvmArgs.set(listOf("-Xmx2g"))                          // avoid MEMORY_ERROR noise on forked minion; raise to 4g for large classpaths
    mutationThreshold.set(60)
    coverageThreshold.set(60)
    testStrengthThreshold.set(60)
}

Key points when proposing:

• Derive targetClasses from rootProject.group (e.g. com.example.foo → "com.example.foo.*"), show the detected value in the diff and ask for confirmation.
• Omit junit5PluginVersion for JUnit 4–only projects. Detect by searching for junit-jupiter in dependencies.
• Pin junit5PluginVersion to the project's JUnit Platform line. Pitest-junit5-plugin bundles its own junit-platform-launcher. If it targets a different Platform than the one in testRuntimeClasspath, the minion dies with OutputDirectoryCreator not available (a silent UNKNOWN_ERROR at the plugin level). Rough matrix (verify against the plugin's release notes before pinning):

  Project junit-platform-launcher	Safe junit5PluginVersion
  1.8.x	1.0.0
  1.9.x	1.1.0
  1.10.x	1.2.0
  1.11.x	1.2.1
  1.12.x	1.2.2
  Newer Platform / Jupiter (6.x / Platform 2.x) may require the pitest configuration to also pin a matching launcher explicitly:

  dependencies { pitest("org.junit.platform:junit-platform-launcher:<version>") }

• Respect existing config — if a pitest { } block already exists, only patch missing fields (e.g. add XML to outputFormats if only HTML is set). Never overwrite thresholds or targetClasses.
• Groovy DSL — emit the equivalent pitest { ... } using = assignment instead of .set().
• Forward tasks.test settings. Pitest's minion does not inherit jvmArgs, systemProperty(...), systemProperties(...), minHeapSize, or maxHeapSize from tasks.test. Inspect the test task and mirror whatever it sets into pitest { jvmArgs = [...] } — system properties become -D entries, heap becomes -Xmx/-Xms. Missing this often manifests as the silent UNKNOWN_ERROR (e.g. JUnit can't instantiate a custom @TestClassOrder because the test task sets it via system property and the minion doesn't see it).

  // If tasks.test has:
  //   maxHeapSize = "4096m"
  //   systemProperty("junit.jupiter.testclass.order.default", "com.example.MyOrderer")
  //
  // Then pitest needs:
  pitest {
      jvmArgs.set(listOf(
          "-Xmx4g",
          "-Dkotlin.jupiter.testclass.order.default=com.example.MyOrderer",
      ))
  }

• Turn on verbose.set(true) during bootstrap. Silent minion crashes produce actionable output only when verbose is on. Can be switched off after the first clean run.

After applying the diff, run ./gradlew pitest once to generate the baseline. First-run failure modes and what they usually mean:

2. Scoped run (on a change)

Derive target classes from git diff and pass them to the wired pitestScope property (see §5 for the wiring). The info.solidsoft.pitest plugin does NOT honor -Ppitest.targetClasses natively — a raw CLI flag with that name will silently be ignored and pitest will run against whatever the pitest { } block has hard-coded.

BASE=$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main)
CHANGED=$(git diff --name-only "$BASE" HEAD -- '*.kt' '*.java' \
  | grep '^src/main/' \
  | sed -E 's#^src/main/(kotlin|java)/##; s#\.(kt|java)$##; s#/#.#g' \
  | paste -sd, -)

if [ -z "$CHANGED" ]; then
  echo "No production classes changed — aborting scoped run."
  exit 1
fi

./gradlew pitest -PpitestScope="$CHANGED"

Aborting on empty match is deliberate — never silently fall back to whole-codebase on a scoped run.

For Groovy-DSL projects, the wiring and flag are identical (same findProperty API).

For multi-module, prefix with the module path: ./gradlew :module-name:pitest -PpitestScope=.... Wire the property in each module's pitest { } block.

3. Interpret survivors

Pitest writes build/reports/pitest/mutations.xml (stable path because timestampedReports.set(false)). Parse it:

# Quick survivor extraction
xmllint --xpath '//mutation[@status="SURVIVED"]' build/reports/pitest/mutations.xml 2>/dev/null

Each <mutation> element carries:

• status — SURVIVED / KILLED / NO_COVERAGE / TIMED_OUT / MEMORY_ERROR / RUN_ERROR / NON_VIABLE
• <sourceFile>, <mutatedClass>, <mutatedMethod>, <lineNumber>, <mutator>, <description>
• <killingTest> — present and populated only for KILLED mutants
• numberOfTestsRun (attribute) — how many tests reached this line

Important: with the default pitest configuration, SURVIVED mutants do not list the names of the tests that reached them — only a count via numberOfTestsRun. The killing-test field in the XML is empty for anything that survived. To identify which test to strengthen, the bootstrap sets fullMutationMatrix.set(true), which adds a <killingTests> / <succeedingTests> matrix to each <mutation> element. Without this flag, the skill can only say "2 tests ran against this line and missed" — it cannot name them.

Side-effect: fullMutationMatrix = true increases report size and slightly increases runtime. Acceptable on scoped runs (one package / one class); consider turning off for whole-codebase CI runs where survivor diagnosis isn't the goal.

exportLineCoverage.set(true) emits a separate coverage.xml listing which tests reach each line. Combined with mutations.xml, you can reconstruct the covering-test set even without fullMutationMatrix, but it requires joining two files. Enabled by default in the bootstrap because it is cheap and useful for the aggregator.

Print a compact table:

STATUS       FILE:LINE              MUTATOR                   DESCRIPTION
SURVIVED     Calculator.kt:42       MATH                      replaced + with -
             covered by: CalculatorTest.`positive number is positive`
SURVIVED     Calculator.kt:43       CONDITIONALS_BOUNDARY     changed > to >=
             covered by: CalculatorTest.`zero is not positive`
NO_COVERAGE  PriceCalc.kt:88        VOID_METHOD_CALLS         removed call to log.debug

Distinguish SURVIVED (test exists but missed the behavior) from NO_COVERAGE (no test reaches the line) and TIMED_OUT (loop mutant caught by pitest's timeout heuristic — not a real survivor).

4. Triage — classify survivors before trying to kill them

This step runs between interpretation and kill-a-survivor. Its job: separate real test-quality gaps from equivalent mutants — mutations that change the bytecode but not the observable behavior in any test a human would reasonably write. Without triage, a kill-a-survivor loop wastes iterations (and, under unattended automation, produces tautological tests) trying to kill things that shouldn't be killed.

Classify each SURVIVED mutant into one of three buckets:

Read the source line at file:line for each survivor and match against these archetypes (expand via the project overlay):

Archetype A — logger-gate conditionals. The line is an if (<numeric> <comparator> <constant>) { ... } whose body contains only calls to loggers (logger.info, .warn, .error, .debug, .trace) or trace/metric sinks. Any ConditionalsBoundary / RemoveConditional_ORDER_* / RemoveConditional_EQUAL_* survivor on such a line is LIKELY_EQUIVALENT.

if (deleted > 0) {          // <— mutations on `> 0` all survive
    logger.info("...")      //   because tests don't capture log output
}

Archetype B — unreachable loop bounds. The line is inside a while / do..while / for condition whose opposing operand is a constant larger than any realistic test input (threshold: default 1000; configurable via overlay). ConditionalsBoundary / RemoveConditional_ORDER_* survivors on such lines are LIKELY_EQUIVALENT — the mutation is theoretically killable but only with infeasible data volumes.

} while (batch.isNotEmpty() && batchCount < maxBatches)  // maxBatches = 10, batch = 10000 rows

Archetype C — null-elvis on non-nullable-in-practice values. The line is a <value> ?: throw <X> pattern where <value> is the non-nullable-by-contract result of a DB primary-key fetch, an Optional.get(), or similar. RemoveConditional_EQUAL_IF / RemoveConditional_EQUAL_ELSE survivors are LIKELY_EQUIVALENT — null is unreachable in any test a DB schema permits.

Kotlin-specific note: pitest's NULL_RETURNS mutator already skips @NotNull-annotated methods, and Kotlin compiles non-nullable return types to @NotNull in bytecode. So null-return mutants only surface on nullable Kotlin returns (Foo?) — archetype C is tuned to that reality and does not over-trigger on plain non-nullable returns.

eventCleanup.hardDeleteEvent(eventId ?: throw IllegalStateException("eventId is null"))
//                           ^-- eventId is jOOQ-fetched primary key, never null in practice

Archetype D — branch-identity (partial-kill residual). An if (cond) { X } else { Y } where X and Y produce the same observable output when cond is false. Classic example: if (list.isNotEmpty()) { repo.fetch(list).associateBy { it.id } } else { emptyMap() } — when list is empty, fetch([]) returns empty and associateBy on empty yields emptyMap, so the IF-branch is indistinguishable from the ELSE-branch for the "empty" case. RemoveConditional_EQUAL_IF mutations on this line survive any test because flipping the branch doesn't change the observable output.

Detecting this archetype statically is hard — it requires knowing that the IF-branch operation is idempotent over the identity input of the ELSE-branch. The script can't match it reliably from source alone. This archetype is surfaced dynamically: when a single strengthened test kills the EQUAL_ELSE mutation on a line but leaves the EQUAL_IF mutation (or vice versa) surviving, the surviving mutation is a strong candidate for LIKELY_EQUIVALENT. Promote it after 2 failed targeted kill attempts (not 3, since the signal is stronger than blind failure) and skip it in future iterations.

Anything that matches no archetype is AMBIGUOUS by default — bias toward ambiguity, not toward equivalence, so real gaps don't get silently excluded.

Implementation. A ready-to-use triage script lives at scripts/triage.py in this skill directory. Run:

python3 .claude/skills/mutation-testing/scripts/triage.py \
  build/reports/pitest/mutations.xml \
  .

It reads the mutations XML, opens each survivor's source line, applies archetypes A/B/C, and writes triage.md + triage.json next to the mutations file.

Output: triage.md (human) and triage.json (machine) next to mutations.xml. Each survivor carries the source pointer, archetype reason, and the names of the tests that covered but didn't kill it (extracted from <succeedingTests> / <coveringTests>). Covering-test names require fullMutationMatrix=true in the bootstrap block; otherwise the script prints a hint to enable it.

Shape:

# Triage of <N> survivors

## LIKELY_KILLABLE (<count>)

- `UserCleanupService.kt:58` **VoidMethodCall** — removed call to deleteS3Files
  - in `UserCleanupService.hardDeleteUser()`
  - reason: no archetype matched
  - covering tests:
    - `UserCleanupServiceTest.hardDeleteUser removes user and all owned data`

## LIKELY_EQUIVALENT (<count>)

- `AnonymousUserCleanupJob.kt:70` **ConditionalsBoundary** — changed > to >=
  - reason: archetype A — logger-gate conditional
- `AnonymousUserCleanupJob.kt:68` **ConditionalsBoundary** — changed < to <=
  - reason: archetype B — loop-bound comparison

The suspected-equivalent.md set is durable — it accumulates across sessions, not per-run. A mutant that's triaged as equivalent once stays in that list unless the source line changes (detected by re-running triage when the line's content differs).

Project overlay extensions. references/project.md can add project-specific archetypes:

• Custom logger types beyond the stdlib set
• Known-non-null value types beyond DB primary keys (e.g. "values returned from MyContext.requireUser()")
• Loop-bound constants specific to the project
• Explicit allowlist of mutants always classified as equivalent (by file:line:mutator key)

Do not skip triage before autoresearch. The mutation-autoresearch skill requires a current triage output as its input — it only iterates LIKELY_KILLABLE. Running autoresearch on the raw survivor list would burn iterations on equivalents and produce tautological tests under pressure.