Prow Job Analysis

Prow Job Analysis

Analyze failures in OpenShift Prow CI jobs. Identify the job type, inspect artifacts, classify the failure, and route to the specialized reference for deep analysis.

Input Format

The user will provide:

1. Prow job URL (required) — Prow UI or gcsweb URL

   • https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id>
   • https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results-public/...

2. Test name (optional) — specific failed test to focus on

3. Flags (optional):

   • --backends <list> — focus disruption analysis on specific backends

Prerequisites

• Python 3.7+: which python3
• jq: which jq
• gcloud CLI (recommended, not required): which gcloud — fastest access to the public bucket (no auth required). Without it, every artifact operation works over plain HTTPS: prow_job_artifact_search.py (stdlib-only list/search/fetch) or curl against https://storage.googleapis.com/test-platform-results-public/....

Investigation Workflow

Step 1: Parse URL and Extract Metadata

1. Find /view/gs/<bucket>/ (Prow UI) or /gcs/<bucket>/ (gcsweb) in the URL. Any non-empty bucket segment is accepted.
2. Extract the object path after the bucket name, then build_id — pattern (\d{10,}) in the path.
3. Construct GCS base: gs://{bucket}/{bucket-path}/ using the URL bucket as-is, including prow-artifact-archive. A private bucket will 403.

Step 2: Fetch prowjob.json

Use the fetch-prowjob-json skill to get job metadata. Extract:

• Job name from .spec.job
• Target from --target= in ci-operator args
• Job state from .status.state
• Refs (org, repo, PR number) from .spec.refs

Step 3: Classify Job Type from Name

Parse the job name to determine the environment and expected failure modes:

Step 4: Download Key Artifacts

mkdir -p .work/prow-job-analysis/{build_id}/logs

# Build log (always)
gcloud storage cp gs://{bucket}/{bucket-path}/build-log.txt \
  .work/prow-job-analysis/{build_id}/logs/ --no-user-output-enabled

# JUnit XML (always — identifies failed tests/steps)
gcloud storage ls "gs://{bucket}/{bucket-path}/artifacts/**/junit*.xml" 2>/dev/null

# Node journals (always, when the job created a cluster) — required input for the
# Step 5 OS-layer check. Gzip-compressed WITHOUT a .gz extension: zcat/zgrep only.
gcloud storage cp -r \
  "gs://{bucket}/{bucket-path}/artifacts/{target}/gather-extra/artifacts/nodes" \
  .work/prow-job-analysis/{build_id}/ --no-user-output-enabled 2>/dev/null || true

Step 5: Classify Failure and Route to Reference

Examine the build log and JUnit results to classify the failure, then consult the appropriate reference file for detailed analysis procedures.

OS-layer evidence check (mandatory for every job, before routing)

Operating-system (RHCOS) layer breakage frequently masquerades as an unrelated product failure: a single RHCOS bump swaps the kernel, cri-o, systemd, NetworkManager, and SELinux policy across the whole cluster at once, so the real cause surfaces as a symptom in some other domain. Before selecting a row from the routing table, complete BOTH steps:

1. Compare runtime versions across boots in the node journals (downloaded in Step 4; gzip-compressed without a .gz extension — plain grep silently matches nothing, use zcat/zgrep):

# Runtime versions per boot. End-of-run snapshots (oc_cmds/nodes, nodes.json)
# show only the final version; changes within the run are visible only here.
zgrep -hE "Starting CRI-O, version|Container runtime initialized" \
  .work/prow-job-analysis/{build_id}/nodes/*/journal | sort | uniq -c

2. Scan the build log, JUnit, oc_cmds (node / clusteroperator status), MachineConfig data, and the journals for these signals:

• NetworkPluginNotReady, or a missing CNI config (/etc/cni/net.d empty / no CNI plugin)
• A ContainerRuntimeVersion change on nodes (cri-o version bump between runs)
• A MachineConfigDaemon (MCD) rendered-config diff touching passwd, files, or units
• Multiple nodes going NotReady after a reboot
• CreateContainerError, RunContainerError, or OCI runtime errors (crun / runc)
• Kernel panic, BUG, Oops, or soft lockup in node journals or the serial console
• avc: denied / SELinux denials
• The same failure spanning multiple unrelated jobs at a payload boundary

If step 1 shows more than one runtime version on any node, or any step-2 signal is present, the RHCOS layer is implicated: still route via the table below using whichever reference matches the surface symptom, but also read operating-system-changes.md alongside it. Never clear the OS layer from end-of-run snapshots alone.