Resolve Muzzle CI

Resolve muzzle CI

Treat the failing task and its first causal exception as the routing evidence. A failed job that happens to download dependencies is not necessarily a muzzle failure.

Match the user's requested scope. For investigation-only requests, gather evidence and report the classification, root cause, and proposed remedy; stop before editing repository files, retrying CI, or creating Jira issues unless those actions are also authorized. Requests to fix or resolve the failure proceed through implementation and verification without another confirmation for already authorized work. Apply the separate CI-retry and Jira authorization rules below.

Confirm and chime in

For GitLab job logs, read CI log access when CLI authentication or job retrieval is needed; the GitHub PR's repository is not necessarily the GitLab CI project.

1. Read the CI job log far enough to identify the full Gradle task path and first causal exception, not only the final Muzzle validation failed wrapper.

2. Confirm that the task is a module :muzzle* task or the runMuzzle aggregate, or that the stack is from datadog.gradle.plugin.muzzle / MuzzleVersionScanPlugin.

3. As soon as it is confirmed, tell the user briefly:

   This is a muzzle CI failure in <task/module>. I am checking whether it is repository
   infrastructure, an artifact/JDK issue, or a real compatibility regression.

   If it is not a muzzle failure, say which task actually failed and stop using this skill.

Before changing a build file, read:

• docs/how_to_work_with_gradle.md
• docs/how_instrumentations_work.md, especially its Muzzle and JApiCmp sections
• .agents/skills/apm-integrations/references/muzzle.md
• the affected module's build.gradle and adjacent versioned modules
• failure signatures and remedies

Classify before editing

Preserve the complete relevant log and CI URL. Record the affected task, module, directive, artifact coordinates, tested version, repository host, and first causal exception.

Use release age only to prioritize investigation. An established version that previously passed suggests checking infrastructure and dependency-graph changes; a newly published version suggests comparing its API and publication with the last passing release. Neither age alone nor a green aggregate rerun justifies a retry, version cap, or skip. See the worked scenarios.

Classify the failure as exactly one of:

• Transient platform/repository failure: timeouts, connection resets, TLS/DNS failures, HTTP 429/5xx, incomplete metadata from MagicMirror/Depot, or several unrelated artifacts failing to download. There is no concrete muzzle mismatch for an already resolved application classpath.
• Defective or unavailable published artifact: the same version or one of its transitive coordinates is persistently absent, has a broken POM/archive, or cannot form a valid classpath, including when checked outside the transient proxy path.
• Compatibility regression: muzzle resolved the application classpath and reports missing or changed classes, methods, fields, flags, or a class-loader mismatch.
• Validation-JDK mismatch: UnsupportedClassVersionError occurs while loading a resolved root or transitive library class that requires a newer JDK than the Muzzle worker.
• Declared-failure mismatch: MUZZLE PASSED ... BUT FAILURE WAS EXPECTED shows that a fail range or inverse expectation is false.
• Helper injection failure: FAILED HELPER INJECTION shows incomplete, misordered, invisible, or incompatible helpers.
• Muzzle tooling/JDK failure: worker, toolchain, module-access, bytecode-parser, or JVM failure; fix the tooling path without changing the library compatibility range.
• Unresolved: the available log is truncated or contradictory. Gather the missing evidence; do not edit a version range merely to make CI green.

For a transient platform failure, do not change repository, version, skip, or exclusion settings. The muzzle resolver already retries range resolution with backoff. If authorized, retry the failed job once after repository recovery, then report an infrastructure incident if the repository remains unhealthy. Stop here; do not create a Jira ticket from this workflow.

For the other classifications, reproduce the smallest task with diagnostic output:

./gradlew :dd-java-agent:instrumentation:<path>:muzzle --stacktrace --info --rerun-tasks

Large ranges are reduced before execution, so a green rerun is not proof that a deterministic mid-range failure disappeared. Rerun the module muzzle task and confirm from its output or report that the exact generated version task ran. If necessary, temporarily narrow a local diagnostic copy of the directive to versions = "[<exact-version>]"; never commit that diagnostic narrowing.

Choose the smallest honest remedy

Choose directly from the observed cause and prove the remedy against the current artifact and API evidence.

The major remedies are:

• Fix the instrumentation when the new release is intended to remain supported and a bounded code change can restore compatibility. Add or update version-specific tests.
• Cap the pass range at the first incompatible version when that version introduces a real API or linkage boundary that this module does not support. Check whether a sibling module should take over; do not skipVersions around an ongoing incompatible line.
• Add skipVersions only for isolated bad releases whose own POM, artifact, or version metadata is defective and where later releases can still be supported.
• Add excludeDependency 'group:module' only for a transitive dependency that is not referenced by the advice, helpers, matcher requirements, explicit muzzle references, or class-loader matcher. Prefer an exact coordinate; use a group wildcard only when the entire group is proven irrelevant.
• Add an exact extraDependency only when it belongs to the library's valid runtime/application classpath but is not present in the resolved graph.
• Ensure a narrowly scoped Gradle repository entry exists when a valid target artifact or required application dependency is intentionally hosted outside the configured repositories. Add extraRepository to the Muzzle directive when version discovery also requires that repository. extraRepository configures Aether version discovery; it does not configure Gradle dependency resolution.
• Correct the target artifact/module when the directive follows a relocated or obsolete coordinate.
• Set per-directive javaVersion = "<N>" when a valid resolved dependency requires that JDK to load. Split at a known JDK boundary when useful and verify both sides of each boundary.
• Change the expected compatibility outcome of a fail block or assertInverse only when MUZZLE PASSED ... BUT FAILURE WAS EXPECTED demonstrates that the expectation is false. Replacing generated inverses with equivalent explicit checks must preserve the expected outcomes and version coverage.
• Repair helper completeness, ordering, linkage, or visibility for FAILED HELPER INJECTION; cap a range only when the helper failure proves a real target-library compatibility boundary.
• Repair Muzzle tooling or CI toolchain provisioning for worker, parser, or JVM failures; do not change the dependency support range.

Generated assertInverse directives do not inherit javaVersion, extraDependency, or extraRepository. When inverse checks need these inputs, replace the generated inverse with explicit fail directives covering the same unsupported ranges, with suitable JDKs, dependencies, and repositories for those versions. Disable assertInverse only after providing equivalent coverage. Ensure Gradle can also resolve the dependencies. Verify that each negative check fails for the intended compatibility mismatch, not a missing classpath input or a repository/JDK failure. Sibling coverage alone does not establish the original module's expected compatibility outcome.

Never broaden support, remove an inverse assertion, or exclude a dependency solely because it makes the task pass. javaVersion changes only the worker JDK; it does not fix library linkage or change the agent, test, or bytecode baselines. Muzzle does not prove that a type named only in a matcher exists; use an explicit muzzle reference or runtime test when that fact matters.

For a symbol mismatch, run the module's printReferences task to trace the generated reference and compare the last passing and first failing artifacts when useful:

./gradlew :dd-java-agent:instrumentation:<path>:printReferences
./gradlew japicmp -Partifact=<group>:<module> -Pbaseline=<last-pass> -Ptarget=<first-fail>

Muzzle validates binary references and helper injection, not behavior. If code or supported runtime semantics change, add a behavior test and exercise a coherent library dependency stack.