Payload Experimental Reverts

Payload Experimental Reverts

This skill experimentally tests medium-confidence candidate PRs by opening draft revert PRs, triggering payload jobs, and evaluating results. It operates in two phases separated by a CI wait period. All state is tracked in the payload results YAML file via the payload-results-yaml skill — no separate tracking file is created.

When to Use This Skill

Use this skill when the /ci:payload-experiment command identifies candidate PRs with medium confidence (score 60-84) that cannot be conclusively attributed to a failure through static analysis alone. The experiment creates real tests to determine causality.

Inputs (passed in-context by the caller):

• results_yaml_path: Path to the payload results YAML file (e.g., ./payload-results-{tag}.yaml)
• candidates: List of medium-confidence PRs to test experimentally, each with:
  • pr_url, pr_number, component, title, confidence_score
  • failing_jobs: List of {job_name, prow_url, is_aggregated, underlying_job_name}

Required Skills

Before starting, you MUST load the following skills (they define output schemas used when updating results):

1. payload-results-yaml — schema for the payload results YAML file
2. payload-autodl-json — schema for the autodl JSON data file

Prerequisites

1. GitHub CLI (gh): Installed and authenticated
2. JIRA MCP: Configured for creating TRT issues (needed in Phase 2 for confirmed causes)
3. Repository Access: User must have push access to their fork of each target repository

Implementation Steps

Phase 1: Set Up Experiments

For each medium-confidence candidate, launch a parallel subagent (do NOT set the model parameter):

1.1: Check for Merge Conflicts

Before opening a revert PR, preemptively check whether the revert will have merge conflicts:

# Clone the repo (shallow for speed)
git clone -b <base_branch> --depth 50 "https://github.com/<org>/<repo>.git" /tmp/experiment-check-<pr_number>
cd /tmp/experiment-check-<pr_number>

# Attempt the revert without committing
git revert -m1 --no-commit <merge_sha>

# Check for conflicts
git status --porcelain

If conflicts exist:

• Append an action entry with type: "experiment", status: "skipped_conflict" to this candidate's actions array
• Skip to the next candidate
• Do NOT attempt to resolve conflicts for experimental reverts

If no conflicts, abort the dry-run revert and proceed:

git revert --abort 2>/dev/null || git checkout -- .

1.2: Open Draft Revert PR

Load the revert-pr skill and follow its workflow with --draft:

• PR URL: the candidate PR
• JIRA ticket: use a placeholder like NO-JIRA (real ticket is created in Phase 2 only for confirmed causes)
• --draft: Create as a draft PR
• --context: "Experimental revert for {stream} {architecture} payload {payload_tag}. Testing whether reverting this PR resolves blocking job failures."
• Do NOT prompt the user for any input

Record the draft revert PR URL.

1.3: Trigger Payload Jobs and Collect Run URLs

Use the trigger-payload-job skill (plugins/ci/skills/trigger-payload-job/SKILL.md) to trigger payload validation jobs on the draft revert PR and collect the resulting URLs. Pass:

• pr_url: The draft revert PR URL
• jobs: The failing_jobs list for this candidate (includes job_name, is_aggregated, underlying_job_name for each job)

1.4: Record Experiment

Use the payload-results-yaml skill to append an action entry to the candidate's actions array:

• type: "experiment"
• status: "pending"
• revert_pr_url, revert_pr_state: "draft", payload_jobs, result_summary: "", jira_key: "", jira_url: ""

See the payload-results-yaml skill for the full schema.

Throttling: Never test more than 5 candidates. If there are more than 5, test only the top 5 by confidence score.

Job triggering limits: Across all experiments combined: trigger at most 5 non-aggregated jobs and at most 1 aggregated job. Prioritize jobs from higher-confidence candidates.

When a candidate is processed but all of its jobs were skipped due to these limits (i.e., none were actually triggered), do NOT leave it with status: "pending". Instead set:

• status: "deferred"
• payload_jobs: one entry per skipped job with command set and test_url, test_prow_url all set to "skipped_due_to_limits"
• result_summary: "All jobs skipped due to cross-experiment triggering limits"

When a candidate has some jobs triggered and some skipped, mark the triggered jobs normally and add entries for skipped jobs with the "skipped_due_to_limits" marker so the record is complete. The action's status should be "pending" in this case (it has real jobs to check).

Candidates beyond the top 5 that were never processed at all should get an action entry with:

• type: "experiment"
• status: "deferred"
• result_summary: "Deferred — exceeded maximum of 5 experimental candidates"

Update Payload Results YAML

After all Phase 1 subagents complete, use the payload-results-yaml skill to update the results file at results_yaml_path with the action entries for each candidate that was processed or deferred.

Update autodl JSON

Use the payload-autodl-json skill's "Update Experiment Status" Phase 1 operation to update the autodl JSON file for each candidate that had a draft revert PR created.

Phase 2: Collect Results and Act

Phase 2 is invoked after a CI wait period (typically 1-4 hours). If the results YAML contains any action entry with type: "experiment" and status: "pending", enter Phase 2. Phase 2 processes only pending experiments — candidates with other statuses are left unchanged.