Analyze Disruption

Analyze Disruption

This skill analyzes disruption events recorded in Prow CI job runs. It downloads interval/timeline data, audit logs, and pod logs, then correlates disruption across backends and job runs to identify root causes.

Prerequisites

1. gcloud CLI Installation

   • Check if installed: which gcloud
   • The test-platform-results-public bucket is public. Do not run gcloud auth login for these artifacts, and do not change the user's gcloud config.
   • download_timelines.py sets CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true on each gcloud subprocess (listing and copying, including parallel workers). That avoids refreshing expired local credentials.
   • Any direct gcloud storage command in this skill uses the same variable. Anonymous access still needs network execution when the command runs in a sandbox.

2. Python 3 (3.7 or later)

Input Format

The user will provide one of the following as input:

Option A — Prow job URLs (direct analysis):

1. One or more Prow job URLs (at least 1)
   • Example: https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn/1983307151598161920

Option B — Grafana disruption dashboard URL (run discovery + analysis):

1. A Grafana disruption dashboard URL — the skill extracts filter parameters, finds matching job runs via Sippy, and presents candidates for the user to select before analysis
   • Example: https://grafana-loki.ci.openshift.org/d/gEdw_aLvk/disruption-for-5-0-os-agnostic?var-platform=gcp&var-backend=host-to-host-new-connections&var-upgrade_type=micro&var-architectures=amd64&var-topologies=ha&var-networks=ovn&var-releases=5.0
   • Recognized by hostname grafana-loki.ci.openshift.org and path starting with /d/
   • Every var-backend value is an exact backend name. Pass that full list as --backends unless the user overrides it. Do not reduce it to one API backend or to a stripped base name

Option C — disruption regression alert text (run discovery + analysis):

1. A pasted DisruptionRegression alert — DisruptionRegressionP50, DisruptionRegressionP50Azure, DisruptionRegressionP75, or DisruptionRegressionP95, or a label block that contains alertname and backend
   • Example labels: backend, platform, upgrade_type, master_nodes_updated, architecture, topology, network, release, delta, feature_set, os, compare_release
   • A link: annotation, when present, is the dashboard URL. Pass the alert text through unchanged. Do not rebuild the URL, and do not drop master_nodes_updated, feature_set, or os
   • With no link, the finder uses the alert labels and a 3-day lookback, which is the regression window these alerts measure

Optional flags (all input options):

2. --backends flag (optional) — comma-separated exact backend names to focus on

   • Example: --backends kube-api-new-connections,oauth-api-new-connections,openshift-api-new-connections
   • If omitted, analyze all backends that show disruption (Option A) or use every Grafana var-backend / alert backend value (Option B or C)

3. --skip-jira flag (optional) — skip the Jira search for known disruption cards

   • By default, the skill searches TRT and OCPBUGS for existing disruption cards after analysis

Bundled Resources

Load these only at the step that needs them — not up front:

• references/run-discovery.md — Grafana URL and alert resolution into Prow job runs (Step 1.5)
• references/artifacts.md — timeline download and optional audit/etcd/PromQL deep dive (Steps 2 and 4)
• references/timeline-analysis.md — parser modes, signal interpretation, and extra diagnostic checks (Steps 3 and 5)
• references/cross-run-comparison.md — multi-run pattern detection and same-job clean comparison (Step 6)
• references/report-guide.md — deep-link shapes, inline linking rules, and report structure (Step 7)
• references/known-issues.md — Jira search and bug filing (Step 8)

Implementation Steps

Step 1: Parse and Validate Input

1. Extract URLs and flags

   • Parse --backends flag if present, split on comma to get backend filter list
   • Parse --skip-jira flag as a boolean option (default: false)
   • Collect all positional URL arguments

2. Detect input type:

   • Alert text: alertname is DisruptionRegressionP50, DisruptionRegressionP50Azure, DisruptionRegressionP75, or DisruptionRegressionP95, or the text is a label block with alertname and backend → proceed directly to Step 1.5 and pass the alert text as --alert-text. Skip item 3 (it runs after Step 1.5 resolves Prow URLs).
   • Grafana URL: hostname is grafana-loki.ci.openshift.org and path starts with /d/ → proceed directly to Step 1.5 to parse URL parameters and find job runs via Sippy. Do NOT fetch the URL, do NOT open or access the dashboard — it is behind SSO. Skip item 3 (it runs after Step 1.5 resolves Prow URLs).
   • Prow URL: any other URL (e.g., prow.ci.openshift.org, gcsweb-ci) → continue to step 3 below
   • Validate at least one URL or one alert is provided

3. Parse each Prow URL to extract bucket path, job name, and build ID

   • Use the same URL parsing logic as the "prow-job-analysis" skill
   • Accept both prow.ci.openshift.org and gcsweb-ci URL formats
   • Extract build_id and job_name from each URL

   Deep-link shapes for the report are in references/report-guide.md. Construct them when writing the report (Step 7), not as a separate links table.

Step 1.5: Resolve a Grafana URL or Alert to Job Runs

Skip this step if the input is Prow job URL(s).

Read references/run-discovery.md (in this skill's directory) and follow it. Pass the Grafana URL or the alert text through unchanged, including every exact backend name. Do not fetch or open the Grafana URL, and do not query Sippy by hand. Do not strip names to a base such as kube-api or substitute derived probes.

The resolved Prow URLs proceed to Step 1, item 3, and then Step 2.

Step 2: Download Artifacts for All Runs

Read references/artifacts.md and follow Timeline download. Write files under .work/disruption-analysis/{date}/.

The downloader uses anonymous access to the public test-platform-results-public bucket. Do not run gcloud auth login. On exit 2, continue with the runs that downloaded. On exit 1, stop and report each run's error.

Step 3: Analyze Interval/Timeline Data

Read references/timeline-analysis.md and follow sections 3.1 through 3.4. Pass exact backend names as --backends. A shortened name is not the selected backend.

Step 4: Deep-Dive Artifact Download (Optional)

Only if the parser output from Step 3 is insufficient for root cause determination, read the Deep-dive artifact download section of references/artifacts.md and follow it. Prefix any direct gcloud storage command with CLOUDSDK_AUTH_DISABLE_CREDENTIALS=true. Do not run gcloud auth login.