Handle PR CI Failure

Handle PR CI failure

Goal

Turn one or more CI failures already attributed to the current PR's own code into either a pushed fix or a clear, evidence-backed report the user can act on. Owning team: @DataDog/agent-devx

Step 0 — Collect the failures to handle

The ideal input is one or more CI triage result blocks (see /triage-ci-failure's output contract: Job, Pipeline SHA, Blame, Failure signature, Evidence, Proposed fix, Incident). When you have these, discard every block whose Blame isn't exactly pr-code — upstream, infra, flake, and inconclusive remain /follow-pr's existing incident/retry path, not this skill's problem. Never treat Incident: none as evidence of pr-code; branch on Blame alone.

In case the input does not follow that format, work with whatever description, log excerpt, or diff you're given. Say plainly that you're missing the structured evidence a full triage would give you, then proceed. Suggest the user runs /triage-ci-failure first.

Either way, group the failures by root cause: several failed jobs (or several sentences of description) sharing one underlying defect are one root cause, not several.

Step 1 — Resolve the autonomy policy

Run the helper script:

python3 .agents/skills/follow-pr/scripts/config.py resolve \
  [--mode autofix|no-autofix|ask] [--max-fix-cycles N] [--policy TEXT]

If the resolved mode is ask, or the script errors, ask the user directly whether this run is autofix or no-autofix before doing anything else.

Any resolved policy text is a classification override, applied on top of Step 2's default criteria — it can broaden them (e.g. "do whatever it takes to make the PR pass" authorizes larger, handwritten fixes the defaults would reject) or narrow them. It can never relax the safety floor in Step 5.

Step 2 — Default safe-fix test

A root cause is autonomous-safe only when all five hold, as modified by any resolved policy text:

1. Clear — the PR caused it, the intended behavior is independently established (an unchanged test, contract, interface, or explicit PR intent), and there's one reasonable correction.
2. Small — one localized correction, not a new implementation or redesign. No new abstraction/API, broad refactor, or multi-part handwritten change. A large generated diff is fine when it comes from a small, reviewed source correction (e.g. one Gazelle-triggering import).
3. Contained — narrow, understood blast radius; no new product, architecture, security, ownership, or shared-CI policy decision.
4. Checkable — concrete before/after evidence; relevant checks pass; the fix doesn't suppress the failing signal. One CI validation push is allowed only when the exact specialized test can't start locally (see Step 4).
5. Clean — the checkout and diff contain only expected work, any generator is stable on a second run, and normal git safeguards stay intact.

This applies equally to unit, E2E, KMT, installer, and platform failures — job family is not a criterion. Only the available evidence and validation fidelity differ.

Examples that satisfy all five:

• A missed rename, inverted conditional, or nil guard fixed against an unchanged test/interface that makes the repair unambiguous.
• Canonical formatter/generator output (gofmt, Buildifier, Gazelle, dda inv tidy, a documented codegen task) reviewed after running it.
• Adding the exact missing needs/rules to a newly introduced leaf CI job, copied from one canonical sibling, when the resolved CI diff touches nothing else.
• Restoring behavior that an unchanged, pre-existing test already required, after a rename or refactor elsewhere in the same PR accidentally broke it.

Examples that fail at least one:

• A multi-method implementation or refactor, even if it makes tests pass — fails Small.
• Changing a golden file or test expectation when that expectation is the only evidence of intent — fails Checkable.
• Retries, sleeps, wider timeouts, skips, suppressions, weaker assertions, or blessing changed bytes under an unchanged artifact identity — fails Checkable (checksum bumps need independent, authenticated provenance, not just "it downloaded fine").
• Adding a nolint/suppression comment to silence a linter finding instead of fixing the underlying issue — fails Checkable.

Step 3 — Classify each root cause

For every grouped root cause from Step 0, decide safe or complex against Step 2. Write down which criterion fails when it's complex — you'll need that for the report in Step 6 either way.

If your harness supports switching to a stronger model mid-task (e.g. an "advisor" or planning mode) and this fix or investigation doesn't feel trivial, consider switching for Steps 4 and 5 — do it where your harness offers it, skip it where it doesn't.

Step 4 — Handle safe root causes

Under no-autofix (or when Step 1 told you the push budget is already exhausted), treat every safe root cause as investigate-only: work through steps 1-6 below to produce and verify a candidate fix, then stop there — leave it uncommitted and describe it in the report exactly like a complex root cause's candidate in Step 5. Steps 7 and the final push only apply under autofix with budget remaining.

For each safe root cause, in this order:

1. Check branch, HEAD, the PR's remote SHA, git status, staged diff, unstaged diff, and untracked files against the snapshot from Step 0. A SHA change caused by a push you already made earlier in this same invocation is expected, not a race — only an external change should block this root cause.
2. Reproduce the failure locally with whatever check actually failed — for example dda inv linter.go --targets=<package> for a lint job, dda inv test --targets=<package> for a unit test, or the job's own e2e/KMT/installer command for those. Don't guess at the command; read it from the failing job's log.
3. Apply one coherent fix for this root cause. Use the repository's own tools (dda inv ..., bazel ...) — never raw go build/go test (see the root AGENTS.md).
4. Run the nearest build/lint/unit checks, then attempt the exact same check that originally failed.
5. If that exact check runs and still fails, this root cause was misclassified: move it to complex (Step 5) and undo any speculative edit for it. If the check can't even start locally, record that limitation — you may still push once for CI validation, but only under autofix, and only if every other pr-code root cause in this batch is also safe.
6. Review the complete diff for this fix. Every changed hunk must map to this root cause; unexplained churn disqualifies it — move it to complex.
7. Under autofix: stage only the explicit paths for this fix, inspect the cached diff, let hooks run normally, and commit with a message describing the actual fix (never "fix CI").

Once every safe root cause has been through the steps above, push once, in one combined push, never once per root cause — and only under autofix, only if every one of them stayed safe and got committed. If any root cause turned out complex, or mode is no-autofix, or the budget was already exhausted, don't push at all: whatever got committed stays local, and Step 5 handles the complex root causes.